# Music Machine Learning - Support Vector Machines (SVM)

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A recap on the [motivation](#motivation) behind Support Vector Machines (SVM)
2. A simple use case for [understanding and fitting SVMs](#fitting) with the scikit library
3. Developing our own [implementation](#implem) of linear SVMs
4. An introuction to [kernels](#improve) in SVMs and how to implement them

## Support Vector Machines (SVM)

In this tutorial, we will cover a more advanced classification algorithm through the use of *Support Vector Machines* (SVMs). The goal is to gain an intuition of how SVMs works and how to use *Gaussian kernel* with SVMs to find out the decision boundary. The implementation proposed here follows a simplified version of the *Sequential Minimal Optimization* (SMO) algorithm for training support vector machines, which is detailed in the following [paper](http://cs229.stanford.edu/materials/smo.pdf). The complete SMO algorithm can then be quite easily implemented following the [full paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-98-14.pdf).

Once again, to simplify your work, we provide the following set of functions that you should find in the `data` folder, and in the `helper_plot` file

  |**File**|*Explanation*|
  |-------:|:------------|
  |`svm_linear.mat`|Example data for a quasi-linear problem|
  |`svm_gaussian.mat`|Example data for a non-linear but well-defined problem|
  |`plot_data`|Allows to plot the set of data points|
  |`visualize_boundary`|Plots a non-linear decision boundary from a SVM model|
  |`plot_svc_decision_function`|Plots a linear decision boundary|

```python
import numpy as np
from scipy import io as sio
from matplotlib import pyplot as plt
from helper_plot import plot_data, visualize_boundary, visualize_boundary_linear, hdr_plot_style, plot_svc_decision_function
```

## Motivation

In the previous course on neural networks, we learned how to train a simple neuron. We saw that this simple model could be interpreted as defining a line, separating the space into two different regions. This allows to consider problems of *discriminative classification*: rather than modeling each class, we simply find a line that divides the classes from each other. As an example of these _linearly separable_ problems, consider the simple case of a classification task, in which the two classes of points are well separated:

```python
from sklearn.datasets import make_blobs
hdr_plot_style()
X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.65)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter');
plt.title("Linear classification example");
```

As we have seen previously, a single neuron classifier would attempt to draw a straight line separating the two sets of data, and thereby create a model for classification, by optimizing the parameters $\theta=\{\mathbf{w}, b\}$ of

$$
\begin{equation}
y=\sigma\left(\sum_{i = 1}^{n}w_{i}.x_{i} + b\right)
\end{equation}
$$

For two dimensional data like that shown here, this is a task we could do by hand. But the major problem that we can directly see, is that there is an infinite number of lines that can perfectly discriminate between the two classes, as exemplified in the next example.

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-w')
plt.xlim(-1, 3.5);
```

Note that these solutions are all valid, and will mostly depend on the (random) initialization of our parameters. Although these all perfectly discriminate between our samples, we can clearly see that they are more or less "optimal" to the problem at hand. So our question today is how could we formalize this problem, and if we could find a formulation to adress and solve this issue.

### Notion of _margin_

The intuition behind comparing different solutions is that rather than estimating only the line between the classes, we can also look at the *margin* of some width defined around those line. In our simple case, it would look like this

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b, d in [(1, 0.65, 0.25), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.15)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-w')
    plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='w', color='#AAAAAA', alpha=0.6)
plt.xlim(-1, 3.5);
```

In *Support Vector Machines (SVMs)*, the goal is to find that optimal model, in which our separating line maximizes this margin. Hence, SVMs are part of a class of models called *maximum margin* estimator.

## Fitting a SVM

More formally the goal of SVMs is that given a set of input vectors $ \mathbf{x} $ separated in two classes (noted $ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $), we would like to find the *optimal boundary* (separating line) between the two classes. To do so, we want to find the separation that gives the *maximal margin* between the two sets. Hence, we can first define the following problem

$$
\begin{equation}
\begin{cases}
\mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}\geq1\\
\mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}\leq-1
\end{cases}
\end{equation}
$$

By setting the variable $ y_{i}={+}/{-}1 $, we can rewrite this problem as

$$
\begin{equation}
y_{i}\left(x_{i}\cdot\mathbf{w}+\mathbf{b}\right)-1\geq0
\end{equation}
$$

The idea is that for some of the input vectors (called *support vectors*), we will have an equality in equation (1), where $ \mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}=1 $ and $ \mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}=-1 $. By substracting those two equation, we see that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification).

### Fitting the SVM with scikit-learn

First, we can see the result of fitting this approach to our original data. To do so, we will use the SVM model found in `scikit-learn` to train an SVM model on this data. To do so, we can use the following code

```python
from sklearn.svm import SVC # "Support vector classifier"
model = SVC(kernel='linear')
model.fit(X, y)
```

    SVC(kernel='linear')

To visualize the results, we will use our quick convenience function that plots the SVM decision boundaries

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model);
```

This separating line is the one that maximizes the margin between our two sets of points.
Notice that some training points are right on the margin. These points are the pivotal elements of this fit, and are known as the *support vectors*, and give the algorithm its name. In `scikit-learn`, we can directly obtain the values of these points, stored in the ``support_vectors_`` attribute of the classifier:

```python
model.support_vectors_
```

    array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]])

The major advantage of SVM is that for the fit, only the position of the support vectors matter; any points further from the margin which are on the correct side do not modify the fit. Technically, this is because these points do not contribute to the loss function used to fit the model, so their position and number do not matter so long as they do not cross the margin. We can see this, for example, if we plot the model learned from the first 50 or 150 points of this dataset

```python
# Plotting SVM trained on subsampled dataset
def plot_svm_subsample(N=10, ax=None):
    X, y = make_blobs(n_samples=200, centers=2, random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    model = SVC(kernel='linear', C=1E10)
    model.fit(X, y)
    ax = ax or plt.gca()
    ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
    ax.set_xlim(-1, 4)
    ax.set_ylim(-1, 6)
    plot_svc_decision_function(model, ax)

fig, ax = plt.subplots(1, 2, figsize=(16, 6))
fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)
for axi, N in zip(ax, [60, 150]):
    plot_svm_subsample(N, axi)
    axi.set_title('N = {0}'.format(N))
```

In the left panel, we see the model and the support vectors for 60 training points.
In the right panel, we have doubled the number of training points, but the model has not changed: the three support vectors from the left panel are still the support vectors from the right panel.
This insensitivity to the exact behavior of distant points is one of the strengths of the SVM model.

## Implementing our own linear SVM

Here, we are going to try to implement our own linear SVM. We recall that, given a set of input vectors $ \mathbf{x} $ separated in two classes ($ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $, we would like to find the *optimal boundary* (separating line) between the two classes. This means that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification). As seen in the slides, we know that this problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

Where the function $\phi(.)$ will be used later to introduce _kernels_, but for now, we will consider that $\phi(x)=x$, which reduces our problem to

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle x_{i},x \right\rangle +b
\end{equation}
$$

we can introduce a kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm and in the first part of this tutorial, we will only consider the *linear* kernel $K\left(x_{i},x\right)=x_{i}\cdot x$.

Now, if we look at the way the SVM model that we trained is implemented in `scikit-learn`, we have the following variables

```python
print(vars(model))
```

    {'decision_function_shape': 'ovr', 'break_ties': False, 'kernel': 'linear', 'degree': 3, 'gamma': 'scale', 'coef0': 0.0, 'tol': 0.001, 'C': 1.0, 'nu': 0.0, 'epsilon': 0.0, 'shrinking': True, 'probability': False, 'cache_size': 200, 'class_weight': None, 'verbose': False, 'max_iter': -1, 'random_state': None, '_sparse': False, 'n_features_in_': 2, 'class_weight_': array([1., 1.]), 'classes_': array([0, 1]), '_gamma': 0.2228546804477968, 'support_': array([18, 25,  7, 46], dtype=int32), 'support_vectors_': array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]]), '_n_support': array([2, 2], dtype=int32), 'dual_coef_': array([[-0.24571733, -0.99437155,  1.        ,  0.24008889]]), 'intercept_': array([3.45657164]), '_probA': array([], dtype=float64), '_probB': array([], dtype=float64), 'fit_status_': 0, 'shape_fit_': (50, 2), '_intercept_': array([-3.45657164]), '_dual_coef_': array([[ 0.24571733,  0.99437155, -1.        , -0.24008889]])}

For the moment, our variables of interest are
- `support_vectors_` : _Support vectors_ (dataset points that support classification)
- `dual_coef_` : The $\alpha_{i} * y_{i}$ (coefficients from the dual formulation) for the _support_vectors_
- `n_support_` : Number of support vectors for each class (ordered)
- `_intercept_` : Value of $b$ in our formulation (bias)

In order to begin our own implementation of the SVM, we are going to start by computing the _decision function_ of the model. We are going to slightly change the model structure, to better reflect the mathematical formulation than `scikit-learn`. Hence, our model structure and all of its corresponding parameters will be defined as

``` Python
model = {};
model["x"];            # Values of _support vectors_ in input data
model["y"];            # Classes labels of _support vectors_
model["n"];            # Number of support for each class
model["b"];            # Value of bias
model["alphas"];       # Value of alphas
model["kernel"];       # Type of kernel
model["w"];            # Weights of vectors in primal formulation
```

In the following, we show how to shift from the `scikit-learn` version to ours

```python
#%% Creating a fake (random) model
our_model = {};
# Values of _support vectors_ in input data
our_model["x"] = model.support_vectors_
# Corresponding classes of the support vectors
our_model["y"] = np.concatenate([[0] * model.n_support_[0], [1] * model.n_support_[1]])
# Number of vectors for each class
our_model["n"] = model.n_support_
# Value of threshold
our_model["b"] = model._intercept_
# Values of alphas
our_model["alphas"] = model._dual_coef_[0]
print(our_model)
```

    {'x': array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]]), 'y': array([0, 0, 1, 1]), 'n': array([2, 2], dtype=int32), 'b': array([-3.45657164]), 'alphas': array([ 0.24571733,  0.99437155, -1.        , -0.24008889])}

Computing the _predictions_ of the SVM is performed by calling `model.decision_function(X)` in `scikit_learn`. We are going to implement this behavior ourselves by coding our own `decision_function` function

``` Python
def decision_function(model, X)
    """
    % Returns a vector of predictions using a trained SVM model.
    % model   : svm model returned from svmTrain.
    % X       : m x n matrix where each example is a row.
    """
    return pred
```

For the time being we are using a _linear kernel_, but we might later use a different type of kernel, so we will directly use a `kernel` function which computes our kernel between the support vectors `x_s` and a dataset `X`.

***

**Exercice**
1. Implement the linear kernel function
1. Complete the decision function based on the dual formulation

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    if (type == 'linear'):
        k = [np.dot(x_s, X) for x_s in x]
    return k

# This function should replicates the scikit-learn one
def decision_function(model, X):

    ######################
    # YOUR CODE GOES HERE

    k=kernel(model["x"],X)
    P=np.zeros(len(k[0]))
    for i in range(len(k)):
        P += model["alphas"][i]*k[i]
    P += model["b"]
    ######################

    return P

# Evaluate the decision function
ax = plt.figure(figsize=(10,8)).gca()
ax.scatter(X[:, 0], X[:, 1], s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(our_model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--']);
ax.scatter(our_model["x"][:, 0], our_model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
```

## Learning phase

In order to perform learning, we will try to maximize the *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. $C$ is a chosen *regularization hyper-parameter*. To ease your work, we will implement only a simplified version of the SMO algorithm which is mainly focused on the optimization aspects. The SMO algorithm selects two $\alpha$ parameters, $\alpha_{i}$ and $\alpha_{j}$ and optimizes the objective value jointly for both these $\alpha_{i,j}$. Finally it adjusts the $b$ parameter based on the new $\alpha_{i,j}$. This process is repeated until the $\alpha_{i,j}$ converge.

### Selecting $\alpha$

The full SMO algorithm use heuristics to choose which $\alpha_{i}$ and $\alpha_{j}$ to optimize (which greatly increase the convergence speed. However, for this simplified version, we will simply iterate on all $\alpha_{i},i=1,\cdots ,m$ and select a random $\alpha_{j}$, to perform the joint optimization of $\alpha_{i}$ and $\alpha_{j}$

### Optimizing $\alpha_{i}$ and $\alpha_{j}$

To jointly optimize the value, we first find bounds $L$ and $H$ such that $L\leq \alpha_{j} \leq H$ holds to satisfy the constraint of the original problem $0\leq \alpha_{i} \leq C$

$$
\begin{array}{ccc}
if y_{i}\neq y_{j}, & L=max\left(0,\alpha_{j}-\alpha_{i}\right), & H=min\left(C,C+\alpha_{j}-\alpha_{i}\right)\\
if y_{i}=y_{j}, & L=max\left(0,\alpha_{i}+\alpha_{j}-C\right), & H=min\left(C,\alpha_{i}+\alpha_{j}\right)
\end{array}
$$

Now we want to find $\alpha_{j}$ so as to maximize the objective function. It can be shown (as in the original paper) that the optimal $\alpha_{j}$ is given by

$$
\alpha_{j}=\alpha_{j}-\frac{y_{j}\left(E_{i}-E_{j}\right)}{\eta}
$$

where

$$
\begin{array}{ccc}
E_{k} & = & f\left(x_{k}\right)-y_{k}\\
\eta & = & 2\left\langle x_{i},x_{j}\right\rangle -\left\langle x_{i},x_{i}\right\rangle -\left\langle x_{j},x_{j}\right\rangle
\end{array}
$$

$E_{k}$ defines the error between the prediction of the SVM on the example $k$ and its desired label. When computing $\eta$, we can also replace the inner product by a kernel function $K$. Finally, we clip the value of $\alpha_{j}$ to lie within the selected bounds.

$$
\alpha_{j}=\begin{cases}
H & if\alpha_{j}>H\\
\alpha_{j} & ifL\leq\alpha_{j}\leq H\\
L & if\alpha_{j}<L
\end{cases}
$$

Now that we have solved $\alpha_{j}$, we can update the value for $\alpha_{i}$ through

$$
\alpha_{i}^{t}=\alpha_{i}^{t-1}+y_{i}y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)
$$

### Computing threshold $b$

Finally, we can compute the threshold $b$ to satisfy the original constraints by selecting

$$
b=\begin{cases}
b_{1} & if0<\alpha_{i}<C\\
b_{2} & if0<\alpha_{j}<C\\
\left(b_{1}+b_{2}\right)/2 & otherwise
\end{cases}
$$

where

$$
b_{1}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{i}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{i},x_{j}\right\rangle
$$
$$
b_{2}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{j}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{j},x_{j}\right\rangle
$$

For the first part of this tutorial, we will compute the main iterations of the algorithm (minimization of the objective function), while relying on a *linear* kernel. This implies that we will only be able to perform linear discrimination. However, remember that the formulation of the SVMs provide an *optimal* and (gloriously) *convex* answer to this problem.

```Python
def svm_train(X, Y, C, kernelFunction, tol, maxIter):
    """
    % Trains an SVM classifier using a simplified SMO algorithm.
    % X       : m x n matrix of m training examples (with n-dimensional features).
    % Y       : column vector of class identifiers
    % C       : standard SVM regularization parameter
    % tol     : tolerance value used for determining equality of floating point numbers.
    % maxIter : number of iterations over the dataset before the algorithm stops.
    """
    return model
```

***

**Exercise**

  1. Update the `svm_train` code to perform the training of a SVM with linear kernel.
  2. Run your `svm_train` function on our linear dataset to obtain similar results as `scikit-learn`
  3. Update the framework to display the decision boundary at each iteration of the algorithm.

***

```python
def kernel(x, X, type='linear'):
    """[summary]

    Args:
        x ([array]): [description]
        X ([array]): [description]
        type (str, optional): [description]. Defaults to 'linear'.

    Returns:
        [type]: [description]
    """
    ######################
    # YOUR CODE GOES HERE

    ######################

    if (type == 'linear'):
        k = [np.dot(x_s, X) for x_s in x]
    return k

def svm_train(X, Y_b, C, kernel_type, tol=1e-3, max_iter=5):
    """
    SVMTRAIN Trains an SVM classifier using a simplified version of the SMO
    # X       : m x n matrix of m training examples (with n-dimensional features).
    # Y       : column vector of class identifiers (0 or 1)
    # C       : standard SVM regularization parameter
    # tol     : tolerance value used for determining equality of floating point numbers.
    # maxIter : number of iterations over the dataset before the algorithm stops.
    """
    # Data parameters
    m, n = X.shape
    # Variables
    alphas = np.zeros((m, 1))
    b, passes, eta, L, H = 0, 0, 0, 0, 0
    E = np.zeros((m, 1))
    Y = Y_b.copy()
    Y[Y == 0] = -1

    # Pre-compute the kernel matrix (in practice, the optimized packages will _not_ do this)
    K = np.array(kernel(X, X.transpose(), kernel_type))

    # Train
    print('Training ...')
    while passes < max_iter:
        # Check that some alphas changed
        num_changed_alphas = 0;
        # Iterative over all alpha_i
        for i in range(m):

            ######################
            # YOUR CODE GOES HERE
            # => Perform one pass of the SMO algorithm for each alpha_i
            for i in range(m):
                j=np.random.randint(m)
                while (j==i):
                    j=np.random.randint(m)

                #bounds L and H
                if Y[i]== Y[j]:
                    L=max(0,alphas[i][0]+alphas[j][0]-C)
                    H=min(C,alphas[i][0]+alphas[j][0])
                else :
                    L=max(0,alphas[i][0]-alphas[j][0])
                    H=min(C,C-alphas[i][0]+alphas[j][0])

                #alpha j computation
                nu=2*np.dot(X[i],X[j].T)-np.dot(X[i],X[i].T)-np.dot(X[j],X[j].T)
                Ei=sum(K[i])+b-Y[i]
                Ej=sum(K[j])+b-Y[j]

                #On garde en mémoire le alpha j précédent
                alphajprev=alphas[j]
                #mise à jour du alpha j
                alphas[j][0]-=Y[j]*(Ei-Ej)/nu
                #clip value
                if alphas[j][0]>H :
                    alphas[j][0]=H
                elif alphas[j][0]<L :
                    alphas[j][0]=L

                #on garde en mémoire le alpha i précédent
                alphaiprev=alphas[i][0]
                #alpha i computation
                alphas[i][0]=alphas[i][0]+Y[i]*Y[j]*(alphajprev-alphas[j])

                #threshold b
                if 0<alphas[i][0]<C:
                    b= b-Ei-Y[i]*(alphaiprev-alphas[i][0])*np.dot(X[i],X[i].T)-Y[j]*(alphajprev-alphas[j][0])*np.dot(X[i],X[j].T)
                elif 0<alphas[j]<C:
                    b= b-Ei-Y[i]*(alphaiprev-alphas[i][0])*np.dot(X[i],X[j].T)-Y[j]*(alphajprev-alphas[j][0])*np.dot(X[j],X[j].T)
                else :
                    b= 1/2*(b-Ei-Y[i]*(alphaiprev-alphas[i][0])*np.dot(X[i],X[i].T)-Y[j]*(alphajprev-alphas[j][0])*np.dot(X[i],X[j].T)+b-Ei-Y[i]*(alphaiprev-alphas[i][0])*np.dot(X[i],X[j].T)-Y[j]*(alphajprev-alphas[j][0])*np.dot(X[j],X[j].T))

            ######################

        if (num_changed_alphas == 0):
            passes = passes + 1
        else:
            passes = 0
    print(' Done !');
    # Save the model
    idx = (alphas > 0).transpose()[0];
    model = {}
    model["x"] = X[idx, :]
    model["y"] = Y[idx]
    model["b"] = b
    model["alphas"] = alphas[idx, 0] * Y[idx]
    model["kernel"] = kernel
    return model

# Training Linear SVM
print('Training Linear SVM ...')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-5       # Tolerance value for thresholds
nb_iterations = 150      # Number of iterations
# Perform the training (try to fiddle the previous values to see effect).
model = svm_train(X, y, C, 'linear', tolerance, nb_iterations);
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

```python
P
```

    array([[ -581.18262766,  -600.11959577,  -619.05656387,  -637.99353198,
             -656.93050009,  -675.86746819,  -694.8044363 ,  -713.74140441,
             -732.67837251,  -751.61534062,  -770.55230873,  -789.48927683,
             -808.42624494,  -827.36321305,  -846.30018115,  -865.23714926,
             -884.17411737,  -903.11108547,  -922.04805358,  -940.98502169,
             -959.9219898 ,  -978.8589579 ,  -997.79592601, -1016.73289412,
            -1035.66986222, -1054.60683033, -1073.54379844, -1092.48076654,
            -1111.41773465, -1130.35470276],
           [ -577.13498905,  -596.07195715,  -615.00892526,  -633.94589337,
             -652.88286147,  -671.81982958,  -690.75679769,  -709.6937658 ,
             -728.6307339 ,  -747.56770201,  -766.50467012,  -785.44163822,
             -804.37860633,  -823.31557444,  -842.25254254,  -861.18951065,
             -880.12647876,  -899.06344686,  -918.00041497,  -936.93738308,
             -955.87435118,  -974.81131929,  -993.7482874 , -1012.6852555 ,
            -1031.62222361, -1050.55919172, -1069.49615982, -1088.43312793,
            -1107.37009604, -1126.30706414],
           [ -573.08735044,  -592.02431854,  -610.96128665,  -629.89825476,
             -648.83522286,  -667.77219097,  -686.70915908,  -705.64612718,
             -724.58309529,  -743.5200634 ,  -762.4570315 ,  -781.39399961,
             -800.33096772,  -819.26793582,  -838.20490393,  -857.14187204,
             -876.07884014,  -895.01580825,  -913.95277636,  -932.88974447,
             -951.82671257,  -970.76368068,  -989.70064879, -1008.63761689,
            -1027.574585  , -1046.51155311, -1065.44852121, -1084.38548932,
            -1103.32245743, -1122.25942553],
           [ -569.03971182,  -587.97667993,  -606.91364804,  -625.85061614,
             -644.78758425,  -663.72455236,  -682.66152047,  -701.59848857,
             -720.53545668,  -739.47242479,  -758.40939289,  -777.346361  ,
             -796.28332911,  -815.22029721,  -834.15726532,  -853.09423343,
             -872.03120153,  -890.96816964,  -909.90513775,  -928.84210585,
             -947.77907396,  -966.71604207,  -985.65301017, -1004.58997828,
            -1023.52694639, -1042.46391449, -1061.4008826 , -1080.33785071,
            -1099.27481881, -1118.21178692],
           [ -564.99207321,  -583.92904132,  -602.86600943,  -621.80297753,
             -640.73994564,  -659.67691375,  -678.61388185,  -697.55084996,
             -716.48781807,  -735.42478617,  -754.36175428,  -773.29872239,
             -792.23569049,  -811.1726586 ,  -830.10962671,  -849.04659481,
             -867.98356292,  -886.92053103,  -905.85749914,  -924.79446724,
             -943.73143535,  -962.66840346,  -981.60537156, -1000.54233967,
            -1019.47930778, -1038.41627588, -1057.35324399, -1076.2902121 ,
            -1095.2271802 , -1114.16414831],
           [ -560.9444346 ,  -579.88140271,  -598.81837081,  -617.75533892,
             -636.69230703,  -655.62927514,  -674.56624324,  -693.50321135,
             -712.44017946,  -731.37714756,  -750.31411567,  -769.25108378,
             -788.18805188,  -807.12501999,  -826.0619881 ,  -844.9989562 ,
             -863.93592431,  -882.87289242,  -901.80986052,  -920.74682863,
             -939.68379674,  -958.62076484,  -977.55773295,  -996.49470106,
            -1015.43166916, -1034.36863727, -1053.30560538, -1072.24257348,
            -1091.17954159, -1110.1165097 ],
           [ -556.89679599,  -575.8337641 ,  -594.7707322 ,  -613.70770031,
             -632.64466842,  -651.58163652,  -670.51860463,  -689.45557274,
             -708.39254084,  -727.32950895,  -746.26647706,  -765.20344516,
             -784.14041327,  -803.07738138,  -822.01434948,  -840.95131759,
             -859.8882857 ,  -878.82525381,  -897.76222191,  -916.69919002,
             -935.63615813,  -954.57312623,  -973.51009434,  -992.44706245,
            -1011.38403055, -1030.32099866, -1049.25796677, -1068.19493487,
            -1087.13190298, -1106.06887109],
           [ -552.84915738,  -571.78612548,  -590.72309359,  -609.6600617 ,
             -628.59702981,  -647.53399791,  -666.47096602,  -685.40793413,
             -704.34490223,  -723.28187034,  -742.21883845,  -761.15580655,
             -780.09277466,  -799.02974277,  -817.96671087,  -836.90367898,
             -855.84064709,  -874.77761519,  -893.7145833 ,  -912.65155141,
             -931.58851951,  -950.52548762,  -969.46245573,  -988.39942383,
            -1007.33639194, -1026.27336005, -1045.21032816, -1064.14729626,
            -1083.08426437, -1102.02123248],
           [ -548.80151877,  -567.73848687,  -586.67545498,  -605.61242309,
             -624.54939119,  -643.4863593 ,  -662.42332741,  -681.36029551,
             -700.29726362,  -719.23423173,  -738.17119983,  -757.10816794,
             -776.04513605,  -794.98210415,  -813.91907226,  -832.85604037,
             -851.79300848,  -870.72997658,  -889.66694469,  -908.6039128 ,
             -927.5408809 ,  -946.47784901,  -965.41481712,  -984.35178522,
            -1003.28875333, -1022.22572144, -1041.16268954, -1060.09965765,
            -1079.03662576, -1097.97359386],
           [ -544.75388015,  -563.69084826,  -582.62781637,  -601.56478448,
             -620.50175258,  -639.43872069,  -658.3756888 ,  -677.3126569 ,
             -696.24962501,  -715.18659312,  -734.12356122,  -753.06052933,
             -771.99749744,  -790.93446554,  -809.87143365,  -828.80840176,
             -847.74536986,  -866.68233797,  -885.61930608,  -904.55627418,
             -923.49324229,  -942.4302104 ,  -961.3671785 ,  -980.30414661,
             -999.24111472, -1018.17808283, -1037.11505093, -1056.05201904,
            -1074.98898715, -1093.92595525],
           [ -540.70624154,  -559.64320965,  -578.58017776,  -597.51714586,
             -616.45411397,  -635.39108208,  -654.32805018,  -673.26501829,
             -692.2019864 ,  -711.1389545 ,  -730.07592261,  -749.01289072,
             -767.94985882,  -786.88682693,  -805.82379504,  -824.76076315,
             -843.69773125,  -862.63469936,  -881.57166747,  -900.50863557,
             -919.44560368,  -938.38257179,  -957.31953989,  -976.256508  ,
             -995.19347611, -1014.13044421, -1033.06741232, -1052.00438043,
            -1070.94134853, -1089.87831664],
           [ -536.65860293,  -555.59557104,  -574.53253915,  -593.46950725,
             -612.40647536,  -631.34344347,  -650.28041157,  -669.21737968,
             -688.15434779,  -707.09131589,  -726.028284  ,  -744.96525211,
             -763.90222021,  -782.83918832,  -801.77615643,  -820.71312453,
             -839.65009264,  -858.58706075,  -877.52402885,  -896.46099696,
             -915.39796507,  -934.33493317,  -953.27190128,  -972.20886939,
             -991.1458375 , -1010.0828056 , -1029.01977371, -1047.95674182,
            -1066.89370992, -1085.83067803],
           [ -532.61096432,  -551.54793243,  -570.48490053,  -589.42186864,
             -608.35883675,  -627.29580485,  -646.23277296,  -665.16974107,
             -684.10670917,  -703.04367728,  -721.98064539,  -740.91761349,
             -759.8545816 ,  -778.79154971,  -797.72851782,  -816.66548592,
             -835.60245403,  -854.53942214,  -873.47639024,  -892.41335835,
             -911.35032646,  -930.28729456,  -949.22426267,  -968.16123078,
             -987.09819888, -1006.03516699, -1024.9721351 , -1043.9091032 ,
            -1062.84607131, -1081.78303942],
           [ -528.56332571,  -547.50029382,  -566.43726192,  -585.37423003,
             -604.31119814,  -623.24816624,  -642.18513435,  -661.12210246,
             -680.05907056,  -698.99603867,  -717.93300678,  -736.86997488,
             -755.80694299,  -774.7439111 ,  -793.6808792 ,  -812.61784731,
             -831.55481542,  -850.49178352,  -869.42875163,  -888.36571974,
             -907.30268784,  -926.23965595,  -945.17662406,  -964.11359217,
             -983.05056027, -1001.98752838, -1020.92449649, -1039.86146459,
            -1058.7984327 , -1077.73540081],
           [ -524.5156871 ,  -543.4526552 ,  -562.38962331,  -581.32659142,
             -600.26355952,  -619.20052763,  -638.13749574,  -657.07446384,
             -676.01143195,  -694.94840006,  -713.88536816,  -732.82233627,
             -751.75930438,  -770.69627249,  -789.63324059,  -808.5702087 ,
             -827.50717681,  -846.44414491,  -865.38111302,  -884.31808113,
             -903.25504923,  -922.19201734,  -941.12898545,  -960.06595355,
             -979.00292166,  -997.93988977, -1016.87685787, -1035.81382598,
            -1054.75079409, -1073.68776219],
           [ -520.46804849,  -539.40501659,  -558.3419847 ,  -577.27895281,
             -596.21592091,  -615.15288902,  -634.08985713,  -653.02682523,
             -671.96379334,  -690.90076145,  -709.83772955,  -728.77469766,
             -747.71166577,  -766.64863387,  -785.58560198,  -804.52257009,
             -823.45953819,  -842.3965063 ,  -861.33347441,  -880.27044251,
             -899.20741062,  -918.14437873,  -937.08134684,  -956.01831494,
             -974.95528305,  -993.89225116, -1012.82921926, -1031.76618737,
            -1050.70315548, -1069.64012358],
           [ -516.42040987,  -535.35737798,  -554.29434609,  -573.23131419,
             -592.1682823 ,  -611.10525041,  -630.04221851,  -648.97918662,
             -667.91615473,  -686.85312284,  -705.79009094,  -724.72705905,
             -743.66402716,  -762.60099526,  -781.53796337,  -800.47493148,
             -819.41189958,  -838.34886769,  -857.2858358 ,  -876.2228039 ,
             -895.15977201,  -914.09674012,  -933.03370822,  -951.97067633,
             -970.90764444,  -989.84461254, -1008.78158065, -1027.71854876,
            -1046.65551686, -1065.59248497],
           [ -512.37277126,  -531.30973937,  -550.24670748,  -569.18367558,
             -588.12064369,  -607.0576118 ,  -625.9945799 ,  -644.93154801,
             -663.86851612,  -682.80548422,  -701.74245233,  -720.67942044,
             -739.61638854,  -758.55335665,  -777.49032476,  -796.42729286,
             -815.36426097,  -834.30122908,  -853.23819718,  -872.17516529,
             -891.1121334 ,  -910.04910151,  -928.98606961,  -947.92303772,
             -966.86000583,  -985.79697393, -1004.73394204, -1023.67091015,
            -1042.60787825, -1061.54484636],
           [ -508.32513265,  -527.26210076,  -546.19906886,  -565.13603697,
             -584.07300508,  -603.00997318,  -621.94694129,  -640.8839094 ,
             -659.82087751,  -678.75784561,  -697.69481372,  -716.63178183,
             -735.56874993,  -754.50571804,  -773.44268615,  -792.37965425,
             -811.31662236,  -830.25359047,  -849.19055857,  -868.12752668,
             -887.06449479,  -906.00146289,  -924.938431  ,  -943.87539911,
             -962.81236721,  -981.74933532, -1000.68630343, -1019.62327153,
            -1038.56023964, -1057.49720775],
           [ -504.27749404,  -523.21446215,  -542.15143025,  -561.08839836,
             -580.02536647,  -598.96233457,  -617.89930268,  -636.83627079,
             -655.77323889,  -674.710207  ,  -693.64717511,  -712.58414321,
             -731.52111132,  -750.45807943,  -769.39504753,  -788.33201564,
             -807.26898375,  -826.20595185,  -845.14291996,  -864.07988807,
             -883.01685618,  -901.95382428,  -920.89079239,  -939.8277605 ,
             -958.7647286 ,  -977.70169671,  -996.63866482, -1015.57563292,
            -1034.51260103, -1053.44956914],
           [ -500.22985543,  -519.16682353,  -538.10379164,  -557.04075975,
             -575.97772785,  -594.91469596,  -613.85166407,  -632.78863218,
             -651.72560028,  -670.66256839,  -689.5995365 ,  -708.5365046 ,
             -727.47347271,  -746.41044082,  -765.34740892,  -784.28437703,
             -803.22134514,  -822.15831324,  -841.09528135,  -860.03224946,
             -878.96921756,  -897.90618567,  -916.84315378,  -935.78012188,
             -954.71708999,  -973.6540581 ,  -992.5910262 , -1011.52799431,
            -1030.46496242, -1049.40193052],
           [ -496.18221682,  -515.11918492,  -534.05615303,  -552.99312114,
             -571.93008924,  -590.86705735,  -609.80402546,  -628.74099356,
             -647.67796167,  -666.61492978,  -685.55189788,  -704.48886599,
             -723.4258341 ,  -742.3628022 ,  -761.29977031,  -780.23673842,
             -799.17370652,  -818.11067463,  -837.04764274,  -855.98461085,
             -874.92157895,  -893.85854706,  -912.79551517,  -931.73248327,
             -950.66945138,  -969.60641949,  -988.54338759, -1007.4803557 ,
            -1026.41732381, -1045.35429191],
           [ -492.1345782 ,  -511.07154631,  -530.00851442,  -548.94548252,
             -567.88245063,  -586.81941874,  -605.75638685,  -624.69335495,
             -643.63032306,  -662.56729117,  -681.50425927,  -700.44122738,
             -719.37819549,  -738.31516359,  -757.2521317 ,  -776.18909981,
             -795.12606791,  -814.06303602,  -833.00000413,  -851.93697223,
             -870.87394034,  -889.81090845,  -908.74787655,  -927.68484466,
             -946.62181277,  -965.55878087,  -984.49574898, -1003.43271709,
            -1022.36968519, -1041.3066533 ],
           [ -488.08693959,  -507.0239077 ,  -525.96087581,  -544.89784391,
             -563.83481202,  -582.77178013,  -601.70874823,  -620.64571634,
             -639.58268445,  -658.51965255,  -677.45662066,  -696.39358877,
             -715.33055687,  -734.26752498,  -753.20449309,  -772.14146119,
             -791.0784293 ,  -810.01539741,  -828.95236552,  -847.88933362,
             -866.82630173,  -885.76326984,  -904.70023794,  -923.63720605,
             -942.57417416,  -961.51114226,  -980.44811037,  -999.38507848,
            -1018.32204658, -1037.25901469],
           [ -484.03930098,  -502.97626909,  -521.91323719,  -540.8502053 ,
             -559.78717341,  -578.72414152,  -597.66110962,  -616.59807773,
             -635.53504584,  -654.47201394,  -673.40898205,  -692.34595016,
             -711.28291826,  -730.21988637,  -749.15685448,  -768.09382258,
             -787.03079069,  -805.9677588 ,  -824.9047269 ,  -843.84169501,
             -862.77866312,  -881.71563122,  -900.65259933,  -919.58956744,
             -938.52653554,  -957.46350365,  -976.40047176,  -995.33743986,
            -1014.27440797, -1033.21137608],
           [ -479.99166237,  -498.92863048,  -517.86559858,  -536.80256669,
             -555.7395348 ,  -574.6765029 ,  -593.61347101,  -612.55043912,
             -631.48740722,  -650.42437533,  -669.36134344,  -688.29831154,
             -707.23527965,  -726.17224776,  -745.10921586,  -764.04618397,
             -782.98315208,  -801.92012019,  -820.85708829,  -839.7940564 ,
             -858.73102451,  -877.66799261,  -896.60496072,  -915.54192883,
             -934.47889693,  -953.41586504,  -972.35283315,  -991.28980125,
            -1010.22676936, -1029.16373747],
           [ -475.94402376,  -494.88099186,  -513.81795997,  -532.75492808,
             -551.69189619,  -570.62886429,  -589.5658324 ,  -608.50280051,
             -627.43976861,  -646.37673672,  -665.31370483,  -684.25067293,
             -703.18764104,  -722.12460915,  -741.06157725,  -759.99854536,
             -778.93551347,  -797.87248157,  -816.80944968,  -835.74641779,
             -854.68338589,  -873.620354  ,  -892.55732211,  -911.49429021,
             -930.43125832,  -949.36822643,  -968.30519453,  -987.24216264,
            -1006.17913075, -1025.11609886],
           [ -471.89638515,  -490.83335325,  -509.77032136,  -528.70728947,
             -547.64425757,  -566.58122568,  -585.51819379,  -604.45516189,
             -623.39213   ,  -642.32909811,  -661.26606621,  -680.20303432,
             -699.14000243,  -718.07697053,  -737.01393864,  -755.95090675,
             -774.88787486,  -793.82484296,  -812.76181107,  -831.69877918,
             -850.63574728,  -869.57271539,  -888.5096835 ,  -907.4466516 ,
             -926.38361971,  -945.32058782,  -964.25755592,  -983.19452403,
            -1002.13149214, -1021.06846024],
           [ -467.84874653,  -486.78571464,  -505.72268275,  -524.65965086,
             -543.59661896,  -562.53358707,  -581.47055518,  -600.40752328,
             -619.34449139,  -638.2814595 ,  -657.2184276 ,  -676.15539571,
             -695.09236382,  -714.02933192,  -732.96630003,  -751.90326814,
             -770.84023624,  -789.77720435,  -808.71417246,  -827.65114056,
             -846.58810867,  -865.52507678,  -884.46204488,  -903.39901299,
             -922.3359811 ,  -941.2729492 ,  -960.20991731,  -979.14688542,
             -998.08385353, -1017.02082163],
           [ -463.80110792,  -482.73807603,  -501.67504414,  -520.61201224,
             -539.54898035,  -558.48594846,  -577.42291656,  -596.35988467,
             -615.29685278,  -634.23382088,  -653.17078899,  -672.1077571 ,
             -691.0447252 ,  -709.98169331,  -728.91866142,  -747.85562953,
             -766.79259763,  -785.72956574,  -804.66653385,  -823.60350195,
             -842.54047006,  -861.47743817,  -880.41440627,  -899.35137438,
             -918.28834249,  -937.22531059,  -956.1622787 ,  -975.09924681,
             -994.03621491, -1012.97318302]])

```python
X[:,0]
```

    array([ 1.35927832,  1.79320251,  1.4709446 ,  1.94742148,  2.09850372,
            2.79587797,  3.27931547,  2.06209253,  2.03694907, -0.13280555,
            1.77595902,  0.16195309,  1.07698591,  1.48924937,  1.53815361,
            1.17976408,  0.72450765,  2.31678959,  0.39920934,  0.90917782,
            1.64295816,  2.28690568,  1.46552358,  1.61816823,  2.52917639,
            2.45161058,  0.64499616, -0.6831733 ,  2.17059451,  2.19018277,
            2.19064626,  0.39421681,  1.26478118, -0.07276352,  1.97257657,
            1.85275838,  2.13776038,  1.0698984 ,  2.35794811,  0.64424674,
            0.9956386 ,  1.59382755,  1.52672244,  1.29792012,  1.00601311,
            1.22945227,  3.02263145,  0.29471065,  2.82147424,  0.75012718])

# Beyond linear problems with _kernels_

For the time being, our SVM is only able to solve _linear_ problems. However, the SVM approach becomes truly powerful is when it is combined with *kernels*. As seen in the slides, the major idea is to project our (non-linearly separable) data into a higher-dimensional space, where it becomes linearly separable (allowing to fit for nonlinear relationships with a linear classifier).

To better understand this idea, we can look at the same non linearly separable data that we discussed in the slides. Here, we try to fit a _linear_ SVM to this non-linear problem.

```python
from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(clf, plot_support=False);
```

It is clear that no linear discrimination will *ever* be able to separate this data. As discussed, we need to think about how we might project the data into a higher dimension such that a linear separator *would* be sufficient. For example, one simple projection we could use would be to compute a *radial basis function* centered on the middle group of points.

```python
r = np.exp(-(X ** 2).sum(1))
```

We can visualize this extra data dimension using a three-dimensional plot

```python
from mpl_toolkits import mplot3d

def plot_3D(elev=30, azim=30, X=X, y=y):
    plt.figure(figsize=(10, 8));
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, edgecolor='w', cmap='winter')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('z')
plot_3D(elev=30, azim=-50, X=X, y=y);
```

We can see that with this additional dimension, the data becomes trivially linearly separable, by drawing a separating plane. Here we had to choose and carefully tune our projection. In general, the need to make such a choice is a problem, as we would like to somehow automatically find the best basis functions to use. One strategy to this end is to compute a basis function centered at *every* point in the dataset, and let the SVM algorithm find the result. This type of basis function transformation is known as a *kernel transformation*, as it is based on a similarity relationship (or kernel) between **each pair of points**.

To see this idea in action first, we will (as previously) use `scikit-learn`, where we can apply _kernelized SVM_ simply by changing our _linear kernel_ to an _RBF (radial basis function)_ kernel, using the ``kernel`` model hyperparameter:

```python
model = SVC(kernel='rbf', C=1E6)
model.fit(X, y)
```

    SVC(C=1000000.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

Once fitted, we can see that our SVM is able to find the largest margin between these two sets of data

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model)
plt.scatter(model.support_vectors_[:, 0], model.support_vectors_[:, 1], s=300, lw=1, facecolors='none');
```

Using this kernelized support vector machine, we learn a suitable nonlinear decision boundary.
This kernel transformation strategy is used often in machine learning to turn fast linear methods into fast nonlinear methods, especially for models in which the kernel trick can be used.

## Implementing the gaussian kernel

As we have seen previously, we know that the complete SVM optimization problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

where we can introduce the kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm. Consequently, the complete *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. The Gaussian kernel (also called *Radial Basis Function (RBF)* kernel) allows to perform the optimization of non-linear classification problems. It is defined as

$$
K\left(x,y\right)=exp\left(\frac{-\left\Vert x-y\right\Vert ^{2}}{\left(2\sigma^{2}\right)}\right)
$$

As seen in the slides, the underlying idea is that instead of trying to solve a non-linear separation problem in a given space, we might first transform the space into a target space in which the problem is linearly separable. Therefore, we transform the space through a kernel (which is directly expressed in the minimization problem) and solve the linear separation problem in the target space. Given the starter code and the previous exercise, you should be able to directly plug this kernel inside the `svmTrain` and `svmPredict` functions.

***

**Exercise**

  1. Update the `kernel` function code to be able to rely on a Gaussian kernel.
  2. Check that you can run the `decision_function` code to rely on this Gaussian kernel.
  2. Run the `svm_train` function on our new (non-linear) dataset to obtain similar results as `scikit-learn`
  3. Once again, display the decision boundary at each iteration of the algorithm.

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    if (type == 'linear'):
        k = ...
    elif kernel == 'gaussian':
        k = ...
    return k

plt.title('Traning Dataset')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-3       # Tolerance value for thresholds
nb_iterations = 20     # Number of iterations
sigma = 0.1            # Variance of Gaussian kernel
# Train the SVM with a Gaussian (RBF) kernel
model = svm_train(X, y, C, 'gaussian', tolerance, nb_iterations)
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

## Supplementary kernels

Now that you have mastered the concept of kernels and updated your code to include the Gaussian kernel, you can augment your classifier and optimizaton problem to include several different kernels as those proposed afterwards.

#### Polynomial kernel.

Intuitively, the polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of these.

$$
K\left(x,y\right)=\left(x^{T}y+1\right)^{d}
$$

#### Sigmoid (tanh) kernel.

This kernel takes two parameters: $a$ and $r$. For $a > 0$, we can view $a$ as a scaling parameter of the input data, and $r$ as a shifting parameter that controls the threshold of mapping. For $a < 0$, the dot-product of the input data is not only scaled but reversed.

$$
K\left(x,y\right)=tanh\left(kx^{T}y+\Theta\right)
$$

***

**Exercise**

  1. Update the `kernel` function code to include these supplementary kernels.
  2. Test both the `decision_function` and `svm_train` codes for these new kernels.

***

```python

```

## Audio application

As seen in the previous tutorial, we have considered only a *binary classification* problem (where elements can only belong to one of two classes). This simplifies the work as we simply need to separate the space between two types of points. However, in real-world problems, we usually want to solve *multi-class* problems with any given number of classes. For this tutorial, we will rely on a simple trick which is to consider that a $n$-class problem can simply be restated as $n$ different binary classification problems. The underlying idea is that each class defines a binary classifier which tells us if a given element is part of this class, or belongs to *any of the other* classes.

***

**Exercise**

  1. Change the training loop to define $n$separate binary problems
  2. For each problem, train an SVM to perform classification.
  3. Evaluate the global classification accuracy of your system.

***

```python

```

```python
from helper_data import import_dataset, compute_transforms, compute_features
class_path = 'data/classification'
# 0.1 - Import the classification dataset
data_struct = import_dataset(class_path, 'classification')
# 0.2 - Pre-process the audio to obtain spectral transforms
data_struct = compute_transforms(data_struct)
# 0.3 - Compute a set of temporal and spectral features
data_struct = compute_features(data_struct)
```


---

Eduard Larrañaga (ealarranaga@unal.edu.co)

---

# Deducción de la Métrica de Schwarzschild
---

## Resumen

En este cuaderno se utiliza [SageMath](https://www.sagemath.org) y [SageManifolds](https://sagemanifolds.obspm.fr/index.html) para deducir la métrica de Schwarzschild a partir de las Ecuaciones de Campo de Einstein.

Nota: Este cuaderno fué creado utilizando SageMath 9.4

```python
version()
```

    'SageMath version 9.4, Release Date: 2021-08-22'

Activamos la visualización matemática via LaTeX,

```python
%display latex
```

## 1.  Definición del Espacio-Tiempo

El espaciotiempo se define como una variedad Lorentziana 4-dimensional,

```python
MU = Manifold(4, 'MU', structure='Lorentzian')
print(MU)
```

    4-dimensional Lorentzian manifold MU

### 1.1. Coordenadas Estandard

Se introducen las coordenadas esfericas (estandard) o **coordenadas de Schwarzschild-Droste**  utilizando el método `.chart()` del objeto variedad`MU`. El argumento del método `.chart()` es una cadena de texto que define cada una de las coordenadas y sus rangos, si son diferentes de $(-\infty, +\infty)$, al igual que el sìmbolo LaTeX que las representará.

Note que las variables de `Python` para cada coordenada son declaradas dentro del operador `<...>`en el lado izquierdo. La variable `X` denotará la carta coordenada.

```python
X.<t,r,th,ph> = MU.chart(r"t r:(0,+oo) th:(0,pi):\theta ph:(0,2*pi):\phi")
X
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(MU,(t, r, {\theta}, {\phi})\right)\]

Es posible acceder a las coordenadas con el formato de un arreglo,

```python
X[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(t, r, {\theta}, {\phi}\right)\]

```python
X[0], X[3]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(t, {\phi}\right)\]

## 2. Tensor Métrico

Para deducir la métrica de Schwarzschild se propone el siguiente elemento de línea que asegura la simetría esférica

\begin{equation}
ds^2 = - A(r) dt^2 + B(r) dr^2 + r^2 d\theta^2 + r^2 \sin^2 \theta d\phi^2.
\end{equation}

Para definir el tensor métrico, primero se declaran las funciones  $A(r)$ y $B(r)$ utilizando `function()`:

```python
A = function('A')(r)
B = function('B')(r)
```

Ahora se utiliza  el método `.metric()` del objeto variedad y se inician sus componentes en la carta `X`, que es la carta por defecto (y única en este caso) para `MU`. El método `.display()` muestra el tensor construido

```python
g = MU.metric()
g[0,0] = -A
g[1,1] = B
g[2,2] = r^2
g[3,3] = (r*sin(th))^2
g.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}g = -A\left(r\right) \mathrm{d} t\otimes \mathrm{d} t + B\left(r\right) \mathrm{d} r\otimes \mathrm{d} r + r^{2} \mathrm{d} {\theta}\otimes \mathrm{d} {\theta} + r^{2} \sin\left({\theta}\right)^{2} \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\]

Las componentes del tensor métrico pueden ser visualizada como un arreglo,

```python
g[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
-A\left(r\right) & 0 & 0 & 0 \\
0 & B\left(r\right) & 0 & 0 \\
0 & 0 & r^{2} & 0 \\
0 & 0 & 0 & r^{2} \sin\left({\theta}\right)^{2}
\end{array}\right)\]

y se pueden acceder a sus componentes en la forma

```python
g[1,1]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}B\left(r\right)\]

### 2.1. Inversa de la Métrica

La inversa del tensor métrico se obtiene mediante el método `.inverse()` o utilizando el método `.up()` aplicado al tensor métrico,

```python
# guu = g.up(g) # Can be defined by rising the indices
guu = g.inverse()
guu
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}g^{-1}\]

```python
guu.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}g^{-1} = -\frac{1}{A\left(r\right)} \frac{\partial}{\partial t }\otimes \frac{\partial}{\partial t } + \frac{1}{B\left(r\right)} \frac{\partial}{\partial r }\otimes \frac{\partial}{\partial r } + \frac{1}{r^{2}} \frac{\partial}{\partial {\theta} }\otimes \frac{\partial}{\partial {\theta} } + \frac{1}{r^{2} \sin\left({\theta}\right)^{2}} \frac{\partial}{\partial {\phi} }\otimes \frac{\partial}{\partial {\phi} }\]

```python
guu[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
-\frac{1}{A\left(r\right)} & 0 & 0 & 0 \\
0 & \frac{1}{B\left(r\right)} & 0 & 0 \\
0 & 0 & \frac{1}{r^{2}} & 0 \\
0 & 0 & 0 & \frac{1}{r^{2} \sin\left({\theta}\right)^{2}}
\end{array}\right)\]

### 2.2. Función Delta

También es posible definir el tensor delta (identidad) $\delta^\mu _\nu = g^\mu _\nu$ utilizando la contracción de índices,

```python
delta = g['_{ab}']*guu['^{bc}']
delta[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1
\end{array}\right)\]

## 3. Ecuaciones de Campo de Einstein

Las Ecuaciones de Campo de Einstein Field Equations están dadas por

\begin{equation}
G_{\mu \nu} = R_{\mu \nu} - \frac{1}{2} g_{\mu \nu} R = \frac{8\pi G}{c^4} T_{\mu \nu}
\end{equation}

Ya que la métrica de Schwarzschild será una solución en el vacio, i.e. $T_{\mu \nu} =0$, las Ecuaciones de Campo se reducen a

\begin{equation}
G_{\mu \nu} = R_{\mu \nu} - \frac{1}{2} g_{\mu \nu} R = 0.
\end{equation}

### 3.1. Tensor de Ricci

Para construir las Ecuaciones de Campo se calculará el tensor de Ricci utilizando el método `.ricci()` de la métrica

```python
Ricci = g.ricci()
Ricci
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mathrm{Ric}\left(g\right)\]

```python
Ricci.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\mathrm{Ric}\left(g\right) = \left( -\frac{r B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} + r A\left(r\right) \frac{\partial\,A}{\partial r} \frac{\partial\,B}{\partial r} - 4 \, A\left(r\right) B\left(r\right) \frac{\partial\,A}{\partial r}}{4 \, r A\left(r\right) B\left(r\right)^{2}} \right) \mathrm{d} t\otimes \mathrm{d} t + \left( \frac{r B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} + {\left(r A\left(r\right) \frac{\partial\,A}{\partial r} + 4 \, A\left(r\right)^{2}\right)} \frac{\partial\,B}{\partial r}}{4 \, r A\left(r\right)^{2} B\left(r\right)} \right) \mathrm{d} r\otimes \mathrm{d} r + \left( \frac{2 \, A\left(r\right) B\left(r\right)^{2} - r B\left(r\right) \frac{\partial\,A}{\partial r} + r A\left(r\right) \frac{\partial\,B}{\partial r} - 2 \, A\left(r\right) B\left(r\right)}{2 \, A\left(r\right) B\left(r\right)^{2}} \right) \mathrm{d} {\theta}\otimes \mathrm{d} {\theta} + \frac{{\left(2 \, A\left(r\right) B\left(r\right)^{2} - r B\left(r\right) \frac{\partial\,A}{\partial r} + r A\left(r\right) \frac{\partial\,B}{\partial r} - 2 \, A\left(r\right) B\left(r\right)\right)} \sin\left({\theta}\right)^{2}}{2 \, A\left(r\right) B\left(r\right)^{2}} \mathrm{d} {\phi}\otimes \mathrm{d} {\phi}\]

```python
Ricci[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
-\frac{r B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} + r A\left(r\right) \frac{\partial\,A}{\partial r} \frac{\partial\,B}{\partial r} - 4 \, A\left(r\right) B\left(r\right) \frac{\partial\,A}{\partial r}}{4 \, r A\left(r\right) B\left(r\right)^{2}} & 0 & 0 & 0 \\
0 & \frac{r B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} + {\left(r A\left(r\right) \frac{\partial\,A}{\partial r} + 4 \, A\left(r\right)^{2}\right)} \frac{\partial\,B}{\partial r}}{4 \, r A\left(r\right)^{2} B\left(r\right)} & 0 & 0 \\
0 & 0 & \frac{2 \, A\left(r\right) B\left(r\right)^{2} - r B\left(r\right) \frac{\partial\,A}{\partial r} + r A\left(r\right) \frac{\partial\,B}{\partial r} - 2 \, A\left(r\right) B\left(r\right)}{2 \, A\left(r\right) B\left(r\right)^{2}} & 0 \\
0 & 0 & 0 & \frac{{\left(2 \, A\left(r\right) B\left(r\right)^{2} - r B\left(r\right) \frac{\partial\,A}{\partial r} + r A\left(r\right) \frac{\partial\,B}{\partial r} - 2 \, A\left(r\right) B\left(r\right)\right)} \sin\left({\theta}\right)^{2}}{2 \, A\left(r\right) B\left(r\right)^{2}}
\end{array}\right)\]

### 3.2. Escalar de Curvatura de Ricci

El escalar de curvatura de Ricci, $R=R^\alpha_\alpha$, se obtiene utilizando el método `.ricci_scalar()` de la métrica

```python
R = g.ricci_scalar()
R.display()
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\begin{array}{llcl} \mathrm{r}\left(g\right):& MU & \longrightarrow & \mathbb{R} \\ & \left(t, r, {\theta}, {\phi}\right) & \longmapsto & \frac{r^{2} B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r^{2} A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} + 4 \, A\left(r\right)^{2} B\left(r\right)^{2} - 4 \, r A\left(r\right) B\left(r\right) \frac{\partial\,A}{\partial r} - 4 \, A\left(r\right)^{2} B\left(r\right) + {\left(r^{2} A\left(r\right) \frac{\partial\,A}{\partial r} + 4 \, r A\left(r\right)^{2}\right)} \frac{\partial\,B}{\partial r}}{2 \, r^{2} A\left(r\right)^{2} B\left(r\right)^{2}} \end{array}\]

---
Ahora declaramos el tensor de Einstein mediante

```python
Einstein = Ricci - g*R/2
Einstein[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
\frac{A\left(r\right) B\left(r\right)^{2} + r A\left(r\right) \frac{\partial\,B}{\partial r} - A\left(r\right) B\left(r\right)}{r^{2} B\left(r\right)^{2}} & 0 & 0 & 0 \\
0 & -\frac{A\left(r\right) B\left(r\right) - r \frac{\partial\,A}{\partial r} - A\left(r\right)}{r^{2} A\left(r\right)} & 0 & 0 \\
0 & 0 & -\frac{r^{2} B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r^{2} A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} - 2 \, r A\left(r\right) B\left(r\right) \frac{\partial\,A}{\partial r} + {\left(r^{2} A\left(r\right) \frac{\partial\,A}{\partial r} + 2 \, r A\left(r\right)^{2}\right)} \frac{\partial\,B}{\partial r}}{4 \, A\left(r\right)^{2} B\left(r\right)^{2}} & 0 \\
0 & 0 & 0 & -\frac{{\left(r^{2} B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r^{2} A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} - 2 \, r A\left(r\right) B\left(r\right) \frac{\partial\,A}{\partial r} + {\left(r^{2} A\left(r\right) \frac{\partial\,A}{\partial r} + 2 \, r A\left(r\right)^{2}\right)} \frac{\partial\,B}{\partial r}\right)} \sin\left({\theta}\right)^{2}}{4 \, A\left(r\right)^{2} B\left(r\right)^{2}}
\end{array}\right)\]

Para separar las ecuaciones diferenciales obtenidas, se utiliza la versión mixta del tensor de Einstein,

\begin{equation}
G^\mu_\nu = R^\mu_\nu - \frac{1}{2} g^\mu_\nu R = 0.
\end{equation}

Para subir el primero de los índices se utiliza el método `.up()`,

```python
Einstein_ud = Einstein.up(g,1)
Einstein_ud[:]
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left(\begin{array}{rrrr}
-\frac{B\left(r\right)^{2} + r \frac{\partial\,B}{\partial r} - B\left(r\right)}{r^{2} B\left(r\right)^{2}} & 0 & 0 & 0 \\
0 & -\frac{A\left(r\right) B\left(r\right) - r \frac{\partial\,A}{\partial r} - A\left(r\right)}{r^{2} A\left(r\right) B\left(r\right)} & 0 & 0 \\
0 & 0 & -\frac{r B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} - 2 \, A\left(r\right) B\left(r\right) \frac{\partial\,A}{\partial r} + {\left(r A\left(r\right) \frac{\partial\,A}{\partial r} + 2 \, A\left(r\right)^{2}\right)} \frac{\partial\,B}{\partial r}}{4 \, r A\left(r\right)^{2} B\left(r\right)^{2}} & 0 \\
0 & 0 & 0 & -\frac{r B\left(r\right) \left(\frac{\partial\,A}{\partial r}\right)^{2} - 2 \, r A\left(r\right) B\left(r\right) \frac{\partial^2\,A}{\partial r ^ 2} - 2 \, A\left(r\right) B\left(r\right) \frac{\partial\,A}{\partial r} + {\left(r A\left(r\right) \frac{\partial\,A}{\partial r} + 2 \, A\left(r\right)^{2}\right)} \frac{\partial\,B}{\partial r}}{4 \, r A\left(r\right)^{2} B\left(r\right)^{2}}
\end{array}\right)\]

## 4. Solución de las Ecuaciones de Campo

La primera de las componentes de las Ecuaciones de Campo es

\begin{equation}
G^0_0 =- \frac{B^2 + r B' - B}{r^2 B^2} = 0
\end{equation}

que corresponde a la ecuación diferencial ordinaria

\begin{equation}
B^2 + r B' - B = 0.
\end{equation}

Se puede resolver esta ecuación con la función `desolve()`

```python
deq = B**2 + r*diff(B,r) - B # Defines the differential equation
B_sol = desolve(deq, B) # Solves the differential equation
B_sol
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}-\log\left(B\left(r\right) - 1\right) + \log\left(B\left(r\right)\right) = C + \log\left(r\right)\]

Redefinimos la constante de integración como $C = -\log(K)$ para despejar la ecuación mediante la función `solve()`,

```python
K = var('K')
solve(- log(B - 1) + log(B) == -log(K) +log(r), B)
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}\left[\log\left(B\left(r\right)\right) = -\log\left(K\right) + \log\left(r\right) + \log\left(B\left(r\right) - 1\right)\right]\]

A aprtir de esta relación podemos obtener

\begin{equation}
\log \left( \frac{B}{B-1} \right) = \log \left(\frac{r}{K} \right)
\end{equation}

\begin{equation}
\frac{B}{B-1}  = \frac{r}{K}
\end{equation}

\begin{equation}
B  = \frac{r}{K} \left( B-1 \right)
\end{equation}

\begin{equation}
B \left( 1-\frac{r}{K} \right) = -\frac{r}{K}
\end{equation}

\begin{equation}
B  = - \frac{\frac{r}{K}}{\left( 1-\frac{r}{K} \right)}
\end{equation}

\begin{equation}
B  =  \frac{1}{\left( 1-\frac{K}{r} \right)}
\end{equation}

---

La segunda componente de las Ecuaciones de Campo es
\begin{equation}
G^1_1 =- \frac{AB - rA' - A }{r^2 AB} = 0
\end{equation}

que correspoinde a la ecuación diferencial ordinaria

\begin{equation}
AB - rA' - A   = 0.
\end{equation}

De nuevo la resolvemos utilizando la función  `desolve()`.

```python
assume(K>0)

deq2 = A*(1/(1-K/r)) - r*diff(A,r) - A == 0
A_sol = desolve(deq2, A, ivar=r)
A_sol
```

\[\newcommand{\Bold}[1]{\mathbf{#1}}C e^{\left(K {\left(\frac{\log\left(-K + r\right)}{K} - \frac{\log\left(r\right)}{K}\right)}\right)}\]

Esta solución da como resultado

\begin{equation}
A = C e^{\log (r-K) - \log (r)}
\end{equation}

\begin{equation}
A = C e^{\log \left( \frac{r-K}{r} \right)}
\end{equation}

\begin{equation}
A = C \left( \frac{r-K}{r} \right)
\end{equation}

\begin{equation}
A = C \left( 1- \frac{K}{r} \right)
\end{equation}

---

Reemplazando las funciones encontradas en el elemento de línea se obtiene

\begin{equation}
ds^2 = - \left( 1- \frac{K}{r} \right) C dt^2 +\left( 1- \frac{K}{r} \right)^{-1} dr^2 + r^2 d\theta^2 + r^2 \sin^2 d\phi^2
\end{equation}

Redefiniendo la coordenada temporal en la forma $c dt^2 \rightarrow dt^2$ y definiendo la constante de itnegración adecuadamente,  $K = \frac{2GM}{c^2}$, se obtiene finalmente la solución de Schwarzschild,

\begin{equation}
ds^2 = - \left( 1- \frac{2GM}{c^2 r} \right) dt^2 +\left( 1- \frac{2GM}{c^2r} \right)^{-1} dr^2 + r^2 d\theta^2 + r^2 \sin^2 d\phi^2.
\end{equation}

### Summary of methods

```
Manifold.chart()
Manifold.metric()
```

```
metric.christoffel_symbols_display()
metric.christoffel_symbols()[]
metric.riemann()
metric.ricci()
```

```
tensor.display()
tensor[:]
tensor.expr()
tensor.up(metric)
tensor.down(metric)
```

```python

```


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 15: Prescott's Real Business Cycle Model II

In this notebook, we continue to examine the centralized version of the model from pages 11-17 in Edward Prescott's article "Theory Ahead of Business Cycle Measurement in the Fall 1986 of the Federal Reserve Bank of Minneapolis' *Quarterly Review* (link to article: [https://www.minneapolisfed.org/research/qr/qr1042.pdf](https://www.minneapolisfed.org/research/qr/qr1042.pdf)). In this notebook, we:

1. Look at the effect of changing $\rho$ (the autoregressive coefficient on log TFP) on the simulated impulse responses of model variables to a TFP shock
2. Compute a stochastic simulation of the model and compute summary statistics.

## Example: Effect of Changing $\rho$ on Impulse Responses

Recall that the equilibrium conditions for Prescott's RBC model are:

\begin{align}
\frac{1}{C_t} & = \beta E_t \left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1}L_{t+1}^{1-\alpha} +1-\delta }{C_{t+1}}\right]\\
\frac{\varphi}{1-L_t} & = \frac{(1-\alpha)A_tK_t^{\alpha}L_t^{-\alpha}}{C_t} \\
Y_t & = A_t K_t^{\alpha}L_t^{1-\alpha}\\
K_{t+1} & = I_t + (1-\delta) K_t\\
Y_t & = C_t + I_t\\
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1}
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$.

The objective is use `linearsolve` to simulate impulse responses to a TFP shock using the following parameter values for the simulation for $\rho = 0.5,0.75,0.9,0.99$. Other parameter values are given in the table below:

| $\sigma$ | $\beta$ | $\varphi$ | $\alpha$ | $\delta $ |
|----------|---------|-----------|----------|-----------|
| 0.006    | 0.99    | 1.7317    | 0.35     |  0.025    |

## Model Preparation

As usual, we recast the model in the form required for `linearsolve`. Write the model with all variables moved to the left-hand side of the equations and dropping the expecations operator $E_t$ and the exogenous shock $\epsilon_{t+1}$:

\begin{align}
0 & = \beta\left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1}L_{t+1}^{1-\alpha} +1-\delta }{C_{t+1}}\right] - \frac{1}{C_t}\\
0 & = \frac{(1-\alpha)A_tK_t^{\alpha}L_t^{-\alpha}}{C_t} - \frac{\varphi}{1-L_t}\\
0 & = A_t K_t^{\alpha}L_t^{1-\alpha} - Y_t\\
0 & = I_t + (1-\delta) K_t - K_{t+1}\\
0 & = C_t + I_t - Y_t\\
0 & = \rho \log A_t - \log A_{t+1}
\end{align}

Remember, capital and TFP are called *state variables* because they're $t+1$ values are predetermined. Output, consumption, and investment are called a *costate* or *control* variables. Note that the model as 5 equations in 5 endogenous variables.

## Initialization, Approximation, and Solution

The next several cells initialize the model in `linearsolve` and then approximate and solve it.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series. CELL PROVIDED
# Note that a value for rho is absent
parameters = pd.Series()

parameters['beta'] = 0.99
parameters['phi'] = 1.7317
parameters['alpha'] = 0.35
parameters['delta'] = 0.025

# Print the model's parameters
print(parameters)
```

    beta     0.9900
    phi      1.7317
    alpha    0.3500
    delta    0.0250
    dtype: float64

```python
# Create a variable called 'sigma' that stores the value of sigma
sigma = 0.006
```

```python
# Create variable called 'varNames' that stores the variable names in a list with state variables ordered first. CELL PROVIDED
var_names = ['a','k','y','c','i','l']

# Create variable called 'shockNames' that stores an exogenous shock name for each state variable. CELL PROVIDED
shock_names = ['e_a','e_k']
```

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero. CELL PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # Euler equation
    mpk = p.alpha*fwd.a*fwd.k**(p.alpha-1)*fwd.l**(1-p.alpha)
    euler_equation = p.beta*(mpk+1-p.delta)/fwd.c - 1/cur.c

    # Labor-labor choise
    mpl = (1-p.alpha)*cur.a*cur.k**p.alpha*cur.l**(-p.alpha)
    labor_leisure = mpl/cur.c - p.phi/(1-cur.l)

    # Production function
    production_function = cur.a*cur.k**p.alpha*cur.l**(1-p.alpha) - cur.y

    # Capital evolution
    capital_evolution = cur.i + (1 - p.delta)*cur.k - fwd.k

    # Market clearing
    market_clearing = cur.c+cur.i - cur.y

    # Exogenous tfp
    tfp_process = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Stack equilibrium conditions into a numpy array
    return np.array([
        euler_equation,
        labor_leisure,
        production_function,
        capital_evolution,
        market_clearing,
        tfp_process
        ])
```

Next, initialize the model using `ls.model` which takes the following required arguments:

* `equations`
* `n_states`
* `var_names`
* `shock_names`
* `parameters`

```python
# Initialize the model into a variable named 'rbc_model'. CELL PROVIDED.
rbc_model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=var_names,
                 shock_names=shock_names,
                 parameters=parameters)
```

### SImulation and Plotting

The objective is to create a $2\times 2$ grid of plots containing the impulse responses of TFP, output, labor, and consumption to a one percent shock to TFP for each of the values for $\rho$: 0.5,0.75,0.9,0.99. Here are the steps that we'll take:

1. Initilize figure and axes for plotting.
2. Iterate over each desired value for $\rho$.
    1. Set `rbc_model.parameters['rho']` equal to current value of $\rho$.
    2. Use `rbc_model.compute_ss()` to compute the steady state with `guess` equal to `[1,4,1,1,1,0.5]`.
    3. Use `rbc_model.approximate_and_solve()` to approximate and solve the model with the current value of $\rho$.
    4. Use `rbc_model.impulse()` to compute the 51 period impulse response to a 0.01 unit shock to TFP in period 5.
    5. Add the computed impulse responses to the axes.

```python
# Create a 12x8 figure
fig = plt.figure(figsize=(12,8))

# Create four axis variables: 'ax1', 'ax2', 'ax3', 'ax4'
ax1 = fig.add_subplot(2,2,1)
ax2 = fig.add_subplot(2,2,2)
ax3 = fig.add_subplot(2,2,3)
ax4 = fig.add_subplot(2,2,4)

# Create an axis equal to the size of the figure. PROVIDED
ax0 = fig.add_subplot(1,1,1)
# Turn off the axis so that the underlying axes are visible. PROVIDED
ax0.set_frame_on(False)
# Hide the x-axis. PROVIDED
ax0.get_xaxis().set_visible(False)
# Hide the y-axis. PROVIDED
ax0.get_yaxis().set_visible(False)

# Create variable called 'rho_values' that stores the desired values of rho
rho_values = [0.5,0.75,0.9,0.99]

# Iterate over the elements of rho_values
for rho in rho_values:

    # Update the value of rho in rbc_model.parameters
    rbc_model.parameters['rho'] = rho

    # Compute the steady state with initial guess equal to [1,4,1,1,1,0.5]
    rbc_model.compute_ss([1,4,1,1,1,0.5])

    # Approximate the model and solve
    rbc_model.approximate_and_solve()

    # Compute the impulse responses to a 0.01 unit shock to TFP
    rbc_model.impulse(T=51,t0=5,shocks=[0.01,0])

    # Add plots of TFP, output, labor, and consumption to ax1, ax2, ax3, and ax4
    ax1.plot(rbc_model.irs['e_a']['a']*100,lw=3,alpha=0.75)
    ax2.plot(rbc_model.irs['e_a']['y']*100,lw=3,alpha=0.75)
    ax3.plot(rbc_model.irs['e_a']['l']*100,lw=3,alpha=0.75)
    ax4.plot(rbc_model.irs['e_a']['c']*100,lw=3,alpha=0.75)

    # Plot the point 0,0 on ax0 with the same line properties used for the other plotted lines and provide a label
    ax0.plot(0,0,lw=3,alpha=0.75,label='$\\rho='+str(rho)+'$')

# Set axis 1 title. PROVIDED
ax1.set_title('TFP')

# Set axis 2 title. PROVIDED
ax2.set_title('Output')

# Set axis 3 title. PROVIDED
ax3.set_title('Labor')

# Set axis 4 title. PROVIDED
ax4.set_title('Consumption')

# Add grid to the axis 1. PROVIDED
ax1.grid()

# Add grid to the axis 2. PROVIDED
ax2.grid()

# Add grid to the axis 3. PROVIDED
ax3.grid()

# Add grid to the axis 4. PROVIDED
ax4.grid()

# Set ax1 y-axis limits to [0,2]. PROVIDED
ax1.set_ylim([0,2])
# Set ax2 y-axis limits to [0,2]. PROVIDED
ax2.set_ylim([0,2])
# Set ax3 y-axis limits to [-0.5,1.25]. PROVIDED
ax3.set_ylim([-0.5,1.5])
# Set ax4 y-axis limits to [-0.5,1.5]. PROVIDED
ax4.set_ylim([-0.5,1.5])

# Add legend below the figure. PROVIDED
legend = ax0.legend(loc='upper center',bbox_to_anchor=(0.5,-0.075), ncol=4,fontsize=15)
```

## Example: Stochastic Simulation

Compute a 401 period stochastic simulation of the model. Set $\rho=0.75$ to match US business cycle data. And set the seed for the simulation to 126.

```python
# Set the value of rho in rbc_model.parameters to 0.75
rbc_model.parameters['rho'] = 0.75

# Compute the steady state with initial guess equal to [1,4,1,1,1,0.5]
rbc_model.compute_ss([1,4,1,1,1,0.5])

# Approximate the model and solve
rbc_model.approximate_and_solve()
```

```python
# Compute the stochastic simulation using the .stoch_sim() method of rbc_model
rbc_model.stoch_sim(T=401,cov_mat=[[sigma**2,0],[0,0]],seed=126)
```

On a single axis, plot the simulated values for output, consumption, investment, and labor.

```python
# Create 12x4 figure. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output, consumption, investment, labor
rbc_model.simulated[['y','c','i','l']].plot(ax=ax,lw=3,alpha=0.75)

# Add legend outside plot. PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.grid()
```

```python
# Print the standard deviations (times 100) of output, consumption, investment, and labor in the simulated data
rbc_model.simulated[['y','c','i','l']].std()*100
```

    y    1.629103
    c    0.646561
    i    5.555957
    l    0.888973
    dtype: float64

```python
# Print the correlations of output, consumption, investment, and labor in the simulated data
rbc_model.simulated[['y','c','i','l']].corr()
```

      y
      c
      i
      l

      y
      1.000000
      0.614215
      0.960996
      0.923899

      c
      0.614215
      1.000000
      0.372012
      0.265520

      i
      0.960996
      0.372012
      1.000000
      0.993686

      l
      0.923899
      0.265520
      0.993686
      1.000000