# Pill 15 - Linear models

Outline:

+ Ordinary least squares regression
+ Logistic regression
+ Generalized additive models
+ Support Vector Machines

# Ordinary least squares regression

We have seen linear models before several times. Let us recall what they are about. The underlying model is a linear one, this is

$$h(x) = a^Tx + b$$

In the context of regression, the use of this model can be associated to the minimization of the squared mean error loss, this is

$$
\begin{align}
\underset{a,b}{\text{minimize}} &\quad \sum_i (y_i - h(x_i;a,b))^2
\end{align}$$

As we saw in other notebooks, this problem can be easily solved using iterative approaches or even using a closed form solution.

# Logistic regression

Logistic regression is a **linear classifier** that is based on maximum likelihood principles to jointly model the classification boundary and the certainty of the fit. It is based on fitting a logistic function to the data.

$$p(x_i|y) = \frac{1}{1+e^{-(a^Tx+b)}}$$

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
beta= 0.5
x = np.linspace(-10,10,100)
p = 1./(1+np.exp(-beta*x))
plt.plot(p)
```

 **EXERCISE: ** What are hypothesis with respect to the probability modeling?

**EXERCISE: ** Change the value of $\beta$ and describe its effect.

The likelihood function can be written as

$$p(x|y) = \prod\limits_{i=1}^N p(x_i|y)^{y_i}(1-p(x_i|y))^{1-y_i}$$

and the goal is to

$$\text{maximize} \quad p(x|y)$$

Taking the logarithm, we can equivalently maximize

$$\text{maximize} \quad \sum_i y_i \ln p(x_i|y) + (1-y_i) \ln(1-p(x_i|x))$$

We can model the probability $p(x_i|x)$ with a logit function, i.e.

$$p(x_i|y) = \ell(x_i) =  \frac{1}{1+e^{-(a^Tx+b)}}$$

In order to optimize this model let us write the derivative of the logit function,

$$\frac{\partial \ell}{\partial a} =  (1+e^{-(a^Tx+b)})^{-2}e^{-(a^Tx+b)}x$$
$$\frac{\partial \ell}{\partial b} =  (1+e^{-(a^Tx+b)})^{-2}e^{-(a^Tx+b)}$$

In terms of the loss function the gradient with respect to the parameters is

$$\nabla_a \mathcal{L} = \sum_i x_i \ell(x_i)(y_i e^{-(a^Tx_i+b)} - (1-y_i) )  $$
$$\nabla_b \mathcal{L} = \sum_i  \ell(x_i)(y_i e^{-(a^Tx_i+b)} - (1-y_i) )  $$

Let us solve this using gradient descent:

```python
%matplotlib inline
import numpy as np
m1 = [0.,0.]
s1 = [[1,-0.9],[-0.9,1]]
m2 = [3.,6.]
s2 = [[1,0],[0,1]]
m3 = [4.,-1.]
s3 = [[1,0.5],[0.5,1]]
print (s3)
c1 = np.random.multivariate_normal(m1,s1,100)
c2 = np.random.multivariate_normal(m2,s2,100)

x= np.r_[c1,c2]
y= np.r_[np.zeros(c1.shape[0]),np.ones(c2.shape[0])]

print (x.shape,y.shape)
import matplotlib.pyplot as plt

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')

plt.gcf().set_size_inches((9,9))
```

```python
#gradient descent

def logit(x,a,b):
    return 1./(1+np.exp(-(np.sum(a*x,axis=1)+b)))

def evaluate_objective(x,y,a,b):
    return np.sum(y*np.log(logit(x,a,b))+(1-y)*np.log((1-logit(x,a,b))))

def partial_a_objective(x,y,a,b):
    partial = x*np.tile((y*np.exp(-(np.sum(a*x,axis=1)+b))*logit(x,a,b)-(1-y)*logit(x,a,b))[:,np.newaxis],(1,2))
    return np.sum(partial,axis=0)

def partial_b_objective(x,y,a,b):
    partial = (y*np.exp(-(np.sum(a*x,axis=1)+b))*logit(x,a,b)-(1-y)*logit(x,a,b))
    return np.sum(partial)

n_iters = 1000
nu=0.01
a = np.random.rand(1,x.shape[1])
b = np.random.rand(1)
conv = []
for i in range(n_iters):
    at = a + nu*partial_a_objective(x,y,a,b) #maximize
    b = b + nu*partial_b_objective(x,y,a,b) #maximize
    a = at
    conv.append(evaluate_objective(x,y,a,b))

plt.plot(conv)
```

```python
print (a,b)

```

    [[ 1.97499493  2.24863925]] [-7.14607427]

```python
#my code
xx,yy = np.meshgrid(np.linspace(-3,7,200),np.linspace(-4,10,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = logit(viz,a,b)

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')
plt.gca().set_xlim([-3.,7.])
plt.gca().set_ylim([-4.,10.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-3,7,-4,10),alpha=0.3, vmin=0, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)))

plt.gcf().set_size_inches((10,10))
```

```python
%matplotlib inline
import numpy as np
m1 = [0.,0.]
s1 = [[1,-0.9],[-0.9,1]]
m2 = [1.,2.]
s2 = [[1,0],[0,1]]
c1 = np.random.multivariate_normal(m1,s1,100)
c2 = np.random.multivariate_normal(m2,s2,100)

x= np.r_[c1,c2]
y= np.r_[np.zeros(c1.shape[0]),np.ones(c2.shape[0])]

print (x.shape,y.shape)
import matplotlib.pyplot as plt

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')

plt.gcf().set_size_inches((9,9))
```

```python
#gradient descent

def logit(x,a,b):
    return 1./(1+np.exp(-(np.sum(a*x,axis=1)+b)))

def evaluate_objective(x,y,a,b):
    return np.sum(y*np.log(logit(x,a,b))+(1-y)*np.log((1-logit(x,a,b))))

def partial_a_objective(x,y,a,b):
    partial = x*np.tile((y*np.exp(-(np.sum(a*x,axis=1)+b))*logit(x,a,b)-(1-y)*logit(x,a,b))[:,np.newaxis],(1,2))
    return np.sum(partial,axis=0)

def partial_b_objective(x,y,a,b):
    partial = (y*np.exp(-(np.sum(a*x,axis=1)+b))*logit(x,a,b)-(1-y)*logit(x,a,b))
    return np.sum(partial)

n_iters = 10000
nu=0.01
a = np.random.rand(1,x.shape[1])
b = np.random.rand(1)
conv = []
for i in range(n_iters):
    at = a + nu*partial_a_objective(x,y,a,b) #maximize
    b = b + nu*partial_b_objective(x,y,a,b) #maximize
    a = at
    conv.append(evaluate_objective(x,y,a,b))

plt.plot(conv)
```

```python
#my code
xx,yy = np.meshgrid(np.linspace(-3,6,200),np.linspace(-4,6,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = logit(viz,a,b)

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')
plt.gca().set_xlim([-3.,6.])
plt.gca().set_ylim([-4.,6.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-3,6,-4,6),alpha=0.3, vmin=0, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)),[0.1,0.5,0.9])

plt.gcf().set_size_inches((9,9))
```

Let us code this model using CVX

# Support vector machines
Support Vector Machines (SVM) is a prototypical example of discriminative learning. In this setting one explicitly assumes a function model class of the boundary. The classical model for SVM is a linear model. SVM is not the only discriminative linear model, e.g. perceptron, logistic classifier, etc. But, it is probably the most complete problem formulation.

Let us first check the intuition behind SVM,

```python
%matplotlib inline
%reset -f
import numpy as np
import matplotlib.pyplot as plt
from IPython.html.widgets import interact

class HLA():
    def __init__(self):
        np.random.seed(1)
        self.X = np.concatenate([1.25*np.random.randn(40,2),5+1.5*np.random.randn(40,2)])
        self.y = np.concatenate([np.ones((40,1)),-np.ones((40,1))])
        plt.scatter(self.X[0:40,0],self.X[0:40,1],color='r')
        plt.scatter(self.X[40:,0],self.X[40:,1],color='b')
        delta = 0.025
        xx = np.arange(-5.0, 10.0, delta)
        yy = np.arange(-5.0, 10.0, delta)
        XX, YY = np.meshgrid(xx, yy)
        Xf = XX.flatten()
        Yf = YY.flatten()
        self.sz=XX.shape
        self.data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);

    def run(self,w0,w1,offset):
        w=np.array([w0,w1])
        w.shape=(2,1)
        Z = self.data.dot(w)+offset
        Z.shape=self.sz
        plt.scatter(self.X[0:40,0],self.X[0:40,1],color='r')
        plt.scatter(self.X[40:,0],self.X[40:,1],color='b')
        plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-5,10,-5,10),alpha=0.3, vmin=-30, vmax=30)
        XX = self.data[:,0].reshape(self.sz)
        YY = self.data[:,1].reshape(self.sz)
        plt.contour(XX,YY,Z,[0])
        fig = plt.gcf()
        fig.set_size_inches(9,9)

def decorator(w0,w1,offset):
    widget_hla.run(w0,w1,offset)

widget_hla = HLA()
interact(decorator, w0=(-10.,10.), w1=(-10.,10.), offset=(-20.,40.));
```

**QUESTION:** Using the former widget, check manually the following configurations:

 $(w_0,w_1,\text{offset}) = (-1.7, -3.1, 10)$

 $(w_0,w_1,\text{offset}) = (-3.7, -0.5, 10.3)$

 $(w_0,w_1,\text{offset}) = (-7.5, -3.2, 28.8)$

Which one of those configuration do you think yields a better boundary? Why?

**INTUITION:** The Support Vector Machine classifer finds the boundary with maximum distance/**margin** to both classes.

Observations:
- It implicitly models the notion of noise. One expects that the boundary with maximum margin will be robust to small perturbations in the data.
- A maximum margin classifier has a unique solution in the separable case.

```python

%reset -f
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from IPython.html.widgets import interact
from sklearn import svm

class svm_example():
    def __init__(self):
        '''Data creation'''
        np.random.seed(1)
        self.X = np.concatenate([1.25*np.random.randn(40,2),5+1.5*np.random.randn(40,2)])
        self.y = np.concatenate([np.ones((40,1)),-np.ones((40,1))])

    def run(self):
        '''Fit a linear SVM'''
        self.clf = svm.SVC(kernel='linear')
        self.clf.fit(self.X,self.y.ravel())

    def display(self):
        '''Display stuff'''
        delta = 0.25
        xx = np.arange(-5.0, 10.0, delta)
        yy = np.arange(-5.0, 10.0, delta)
        XX, YY = np.meshgrid(xx, yy)
        Xf = XX.flatten()
        Yf = YY.flatten()
        sz=XX.shape
        data = np.concatenate([Xf[:,np.newaxis],Yf[:,np.newaxis]],axis=1);
        Z=self.clf.decision_function(data)
        Z.shape=sz
        plt.scatter(self.X[0:40,0],self.X[0:40,1],color='r')
        plt.scatter(self.X[40:,0],self.X[40:,1],color='b')
        plt.imshow(Z, interpolation='bilinear', origin='lower', extent=(-5,10,-5,10),alpha=0.3, vmin=-3, vmax=3)
        XX = data[:,0].reshape(sz)
        YY = data[:,1].reshape(sz)
        plt.contour(XX,YY,Z,[-1,0,1],colors=['b','k','r'])
        fig = plt.gcf()
        fig.set_size_inches(9,9)
        print ('Number of support vectors: ' + str(np.sum(self.clf.n_support_)))
        plt.scatter(self.clf.support_vectors_[:, 0],
           self.clf.support_vectors_[:, 1],
           s=120,
           facecolors='none',
           linewidths=2,
           zorder=10)
        print ('(w0,w1) = ' + str(10*self.clf.coef_[0]))
        print ('offset = ' + str(10*self.clf.intercept_[0]))
        return XX,YY,Z

c = svm_example()
c.run()
XX,YY,Z=c.display()
```

Observe that there is a critical subset of data points. These are called **Support Vectors**. If any of those points disappear the boundary changes.  The decision boundary depends on the support vectors, thus we have to store them in our model.

Check the intuition in 3D:

```python
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
np.random.seed(1)
X = np.concatenate([1.25*np.random.randn(40,2),5+1.5*np.random.randn(40,2)])
y = np.concatenate([np.ones((40,1)),-np.ones((40,1))])
def control3D(elevation,azimuth):
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    fig.set_size_inches(12,12)
    ax.plot_surface(XX,YY,Z,cmap=cm.coolwarm,alpha=0.3,linewidth=0)
    ax.scatter(X[0:40,0],X[0:40,1],1,color='r')
    ax.scatter(X[40:,0],X[40:,1],-1,color='b')
    ax.contour(XX,YY,Z,[-1,0,1],colors=['b','k','r'])
    ax.view_init(elev=elevation, azim=azimuth)

#Ipython 2.0
interact(control3D,elevation=(0.,90.),azimuth=(0,360))
#Ipython 1.1
#elevation = 45
#azimuth = 180
#control3D(elevation,azimuth)
```

**QUESTIONS: **

 Set the azimuth to $113$ and elevation to $0$. Observe the data points and the relative position of the hyperplane.

 Change the elevation to $90$. Describe this projection.

### 3.1.1 Geometry of the hyperplane
A hyperplane in ${\bf R}^d$ is defined as an affine combination of the variables: $\pi\equiv a^Tx + b = 0$.

Features:

+ A hyperplane splits the space in two half-spaces. The evaluation of the equation of the hyperplane on any element of one of the half-space is a positive value. It is a negative value for all the elements in the other half-space.
+ The distance of a point $x \in{\bf R}^d$ to the hyperplane $\pi$ is
$$d(x,\pi)=\frac{a^Tx+b}{\|a\|_2}$$

### 3.1.2 Modeling the separating hyperplane
Given a binary classification problem with training data $\mathcal{D}=\{(x_i,y_i)\},\; i=1\dots N, \; y_i\in\{+1,-1\} $. Consider $\mathcal{S} \subseteq \mathcal{D}$ the subset of all data points belonging to class $+1$, $\mathcal{S}=\{x_i | y_i=+1\}$, and $\mathcal{R}=\{x_i | y_i=-1\}$ its complement.

Then the problem of finding a separating hyperplane consists of fulfilling the following constraints

$$a^Ts_i+b>0\; \text{and}\; a^Tr_i+b<0 \quad \forall s_i\in\mathcal{S}, r_i\in\mathcal{R}.$$

Note the strict inequalities in the formulation. Informally, we can consider the smallest satisfied constraint. And observe that the rest must be satisfied with a larger value. Thus, we can arbitrarily set that value to 1 and rewrite the problem as $$a^Ts_i+b\geq 1\; \text{and}\; a^Tr_i+b\leq -1.$$

This is a *feasibility problem* and it is usually written in the following way in optimization standard notation

$$
\begin{align}
\text{minimize} & 1\\
\text{subject to} & a^T r_i + b \leq -1,\; \forall r_i \in \mathcal{R}\\
& a^T s_i + b \geq 1\; \forall s_i \in \mathcal{S}
\end{align}
$$

or in a compact way

$$
\begin{align}
\text{minimize} & 1\\
\text{subject to} & y_i (a^T x_i + b) \geq 1,\; \forall x_i \in \mathcal{D}\\
\end{align}
$$

The solution of this problem is not unique, e.g. remember all the parameters of the 'Human Learning Algorithm'.

```python
%matplotlib inline
import numpy as np
m1 = [0.,0.]
s1 = [[1,-0.9],[-0.9,1]]
m2 = [3.,6.]
s2 = [[1,0],[0,1]]
m3 = [4.,-1.]
s3 = [[1,0.5],[0.5,1]]
print (s3)
c1 = np.random.multivariate_normal(m1,s1,100)
c2 = np.random.multivariate_normal(m2,s2,100)

x= np.r_[c1,c2]
y= np.r_[-np.ones(c1.shape[0]),np.ones(c2.shape[0])]

print (x.shape,y.shape)
import matplotlib.pyplot as plt

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')

plt.gcf().set_size_inches((9,9))
```

```python
from cvxpy import *
import numpy

d = x.shape[1]
N = x.shape[0]

# Construct the problem.
a = Variable(d)
b = Variable()
error = 1.
obj = Minimize(error)
constraints = [mul_elemwise(y,x*a+b)>=1]
prob = Problem(obj,constraints)
prob.solve()
```

    1.0

```python
prob.status
```

    'optimal'

```python
a.value, b.value
```

    (matrix([[ 2.18098634],
             [ 2.58581768]]), -7.4021516960174134)

 **EXERCISE: ** Fill the blank with the model function to predict the boundary

```python
#your code
xx,yy = np.meshgrid(np.linspace(-3,6,200),np.linspace(-4,6,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = # PUT HERE YOUR CODE

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')
plt.gca().set_xlim([-3.,6.])
plt.gca().set_ylim([-4.,6.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-3,6,-4,6),alpha=0.3, vmin=-1, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)),[0.])

plt.gcf().set_size_inches((9,9))
```

### 3.1.3 The maximum margin hyperplane

Selecting the maximum margin hyperplane requires to add a new constraint to our problem. Remember from the geometry of the hyperplane that the distance of any point to a hyperplane is given by $d(x,\pi)=\frac{a^Tx+b}{\|a\|_2}$.

Recall that we want positive data to be beyond value 1 and negative data below -1. Thus, what is the distance value we want to maximize?

The positive point closest to the boundary is at $1/\|a\|_2$ and the negative point closest to the boundary data point is also at $1/\|a\|_2$. Thus data points from different classes are at least $2/\|a\|_2$ apart.

Recall that our goal is to find the separating hyperplane with maximum margin, i.e. with maximum distance among elements from different classes. Thus, we can complete the former formulation with our last requirement as follows

$$
\begin{align}
\text{maximize} & \quad 2/\|a\|_2 \\
\text{subject to} & \quad y_i (a^T x_i + b) \geq 1,\; \forall x_i \in \mathcal{D}\\
\end{align}
$$

or equivalently,

$$
\begin{align}
\text{minimize} & \quad \|a\|_2/2 \\
\text{subject to} & \quad y_i (a^T x_i + b) \geq 1,\; \forall x_i \in \mathcal{D}\\
\end{align}
$$

This formulation has a solution as long as the problem is linearly separable.

**EXERCISE: ** Code the hard margin SVM.

```python
%matplotlib inline
import numpy as np
m1 = [0.,0.]
s1 = [[1,-0.9],[-0.9,1]]
m2 = [3.,6.]
s2 = [[1,0],[0,1]]
m3 = [4.,-1.]
s3 = [[1,0.5],[0.5,1]]
print (s3)
c1 = np.random.multivariate_normal(m1,s1,100)
c2 = np.random.multivariate_normal(m2,s2,100)

x= np.r_[c1,c2]
y= np.r_[-np.ones(c1.shape[0]),np.ones(c2.shape[0])]

print (x.shape,y.shape)
import matplotlib.pyplot as plt

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')

plt.gcf().set_size_inches((9,9))
```

```python
#Your code here

from cvxpy import *
import numpy

x = x[0:,:]
y = y[0:]

d = x.shape[1]
N = x.shape[0]

# Construct the problem.
# Declare variables
objective= ###########
constraints= ###########

prob = Problem(objective,constraints)
prob.solve()

```

```python
a.value
```

    matrix([[ 0.27103272],
            [ 0.40360088]])

```python
#my code
xx,yy = np.meshgrid(np.linspace(-3,6,200),np.linspace(-4,6,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = np.dot(viz,a.value)+b.value

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')
plt.gca().set_xlim([-3.,6.])
plt.gca().set_ylim([-4.,6.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-3,6,-4,6),alpha=0.3, vmin=-1, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)),[-1.,0.,1.])

plt.gcf().set_size_inches((9,9))
```

### 3.1.4 Dealing with the non-separable case

In order to deal with misclassifications, we are going to introduce a new set of variables $\xi_i$, that represents the amount of violation in the $i-th$ constraint. If the constraint is already satisfied, then $\xi_i=0$, and $\xi_i>0$ otherwise. Because $\xi_i$ is related to the errors, we would like to keep this amount as close so zero as possible. This makes us introduce a element in the objective trading-off with the maximum margin.

The new model becomes

$$
\begin{align}
\text{minimize} & \|a\|_2/2 + C \sum\limits_{i=1}^N \xi_i\\
\text{subject to} & y_i (a^T x_i + b) \geq 1 - \xi_i,\; i=1\dots N\\
& \xi_i\geq 0
\end{align}
$$

where $C$ is the trade-off parameter that roughly balances margin and misclassification rate. This formulation is also called **soft-margin SVM**.

```python
%matplotlib inline
import numpy as np
m1 = [0.,0.]
s1 = [[1,-0.9],[-0.9,1]]
m2 = [1.,2.]
s2 = [[1,0],[0,1]]
c1 = np.random.multivariate_normal(m1,s1,100)
c2 = np.random.multivariate_normal(m2,s2,100)

x= np.r_[c1,c2]
y= np.r_[-np.ones(c1.shape[0]),np.ones(c2.shape[0])]

print (x.shape,y.shape)
import matplotlib.pyplot as plt

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')

plt.gcf().set_size_inches((9,9))
```

**EXERCISE: ** Run the code for the hard margin SVM. Describe the result.

```python
#Your code here

from cvxpy import *
import numpy

d = x.shape[1]
N = x.shape[0]

# Construct the problem.
# Declare variables
objective= ###########
constraints= ###########

prob = Problem(objective,constraints)
prob.solve()
prob.status

```

**EXERCISE: ** Code the soft margin SVM. Describe the result when changing parameter C.

```python
#your code
#my code
from cvxpy import *
import numpy

d = x.shape[1]
N = x.shape[0]
C = 100.

# Construct the problem.
# Declare variables
objective= ###########
constraints= ###########

prob = Problem(objective,constraints)
prob.solve()
prob.status

```

```python
xx,yy = np.meshgrid(np.linspace(-3,6,200),np.linspace(-4,6,200))
viz=np.c_[xx.ravel(),yy.ravel()]

z = np.dot(viz,a.value)+b.value

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')
plt.gca().set_xlim([-3.,6.])
plt.gca().set_ylim([-4.,6.])
plt.imshow(z.reshape((200,200)), interpolation='bilinear', origin='lower', extent=(-3,6,-4,6),alpha=0.3, vmin=-1, vmax=1)
plt.contour(xx,yy,z.reshape((200,200)),[-1.,0.,1.])

plt.gcf().set_size_inches((9,9))
```

**EXERCISE: ** Consider the formulation of the soft-margin SVM in a separable case. If one sets parameter $C$ to 0, what is the resutl? do we recover the hard-margin SVM?

**Take home ideas:**

 Classical SVM fits a hyperplane separating boundary.

 The hyperplane is defined to achieve the maximum margin.

 If the problem is not linearly separable a new term related to the misclassification performance is introduced that trades-off with the margin. This trade-off is governed by parameter $C$ (or $\nu$ in $\nu$-SVM).

**DELIVERABLE: ** We want to modify the soft-margin SVM to be able to accomodate unbalanced data. Describe how to change the formulation such that we can control the balance of the errors from each class. Code an example and apply it to the following problem.

```python
%matplotlib inline
import numpy as np
m1 = [0.,0.]
s1 = [[0.05,0],[0,0.05]]
m2 = [1.,2.]
s2 = [[1,0],[0,1]]
c1 = np.random.multivariate_normal(m1,s1,20)
c2 = np.random.multivariate_normal(m2,s2,300)

x= np.r_[c1,c2]
y= np.r_[-np.ones(c1.shape[0]),np.ones(c2.shape[0])]

print (x.shape,y.shape)
import matplotlib.pyplot as plt

plt.plot(c1[:,0],c1[:,1],'r.')
plt.plot(c2[:,0],c2[:,1],'b.')

plt.gcf().set_size_inches((9,9))
```

```python

```

```python

```

```python

```

```python

```


```python
import sympy as sp
```

```python
x = sp.symbols('x')
expr = x**2 - x + 1
diff = sp.Derivative(expr, x)
diff
```

$\displaystyle \frac{d}{d x} \left(x^{2} - x + 1\right)$

```python
diff.doit()
```

$\displaystyle 2 x - 1$

At the minimum: $x = \frac{1}{2}$

Function value at at minimum:

```python
0.5**2-0.5+1
```

    0.75

```python
import numpy as np
import tensorflow as tf
```

```python
tf.__version__
```

    '1.14.0'

```python
A = tf.placeholder(tf.float32)
v = tf.placeholder(tf.float32)
op = tf.matmul(A,v)
```

```python
with tf.Session() as session:
    output = session.run(op, feed_dict={A:np.random.random((3,3)),v:np.random.random((3,1))})
```

```python
print(output)
```

    [[0.6251757 ]
     [0.98640007]
     [0.24678826]]

```python
u = tf.Variable(20.0)
```

```python
func = u*u - u + 1
```

```python
opt = tf.train.GradientDescentOptimizer(0.22).minimize(func)
```

```python
init = tf.global_variables_initializer()
```

```python
with tf.Session() as session:
    session.run(init)
    for i in range(15):
        minimum = session.run(opt)
        print("iteration: %d - func_value: %.2f - u_value: %0.2f" % (i, func.eval(), u.eval()))
```

    iteration: 0 - func_value: 120.00 - u_value: 11.42
    iteration: 1 - func_value: 38.15 - u_value: 6.62
    iteration: 2 - func_value: 12.48 - u_value: 3.92
    iteration: 3 - func_value: 4.43 - u_value: 2.42
    iteration: 4 - func_value: 1.90 - u_value: 1.57
    iteration: 5 - func_value: 1.11 - u_value: 1.10
    iteration: 6 - func_value: 0.86 - u_value: 0.84
    iteration: 7 - func_value: 0.79 - u_value: 0.69
    iteration: 8 - func_value: 0.76 - u_value: 0.61
    iteration: 9 - func_value: 0.75 - u_value: 0.56
    iteration: 10 - func_value: 0.75 - u_value: 0.53
    iteration: 11 - func_value: 0.75 - u_value: 0.52
    iteration: 12 - func_value: 0.75 - u_value: 0.51
    iteration: 13 - func_value: 0.75 - u_value: 0.51
    iteration: 14 - func_value: 0.75 - u_value: 0.50


## Regularization

```python
from IPython.display import Image
Image('../../../python_for_probability_statistics_and_machine_learning.jpg')
```

We have referred to regularization in earlier sections, but we
want to develop this important idea more fully. Regularization is
the mechanism by which we navigate the bias/variance trade-off.
To get started, let's consider a classic constrained least
squares problem,

$$
\begin{aligned}
& \underset{\mathbf{x}}{\text{minimize}}
& & \Vert\mathbf{x}\Vert_2^2 \\
& \text{subject to:}
& & x_0 + 2 x_1 = 1
\end{aligned}
$$

 where $\Vert\mathbf{x}\Vert_2=\sqrt{x_0^2+x_1^2}$ is the
$L_2$ norm.  Without the constraint, it would be easy to minimize
the objective function --- just take $\mathbf{x}=0$. Otherwise,
suppose we somehow know  that $\Vert\mathbf{x}\Vert_2<c$, then
the locus of points defined by this inequality is the circle in
[Figure](#fig:regularization_001). The constraint is the line in
the same figure. Because every value of $c$ defines a circle, the
constraint is satisfied  when the circle touches the line. The
circle can touch the line at many different points, but we are
only interested in the smallest such circle because this is a
minimization problem.  Intuitively, this means that we *inflate* a
$L_2$  ball at the origin and stop when it just touches the
contraint.  The point of contact is our $L_2$ minimization
solution.

The solution of the constrained $L_2$ minimization problem is at the point
where the constraint (dark line) intersects the $L_2$ ball (gray circle)
centered at the origin. The point of intersection is indicated by the dark
circle. The two neighboring squares indicate points on the line that are close
to the solution.

We can obtain the same result using the method of Lagrange
multipliers. We can rewrite the entire $L_2$ minimization problem
as one objective function using the Lagrange multiplier,
$\lambda$,

$$
J(x_0,x_1,\lambda) = x_0^2+x_1^2 + \lambda (1-x_0-x_1)
$$

 and solve this as an ordinary function using calculus. Let's
do this using Sympy.

```python
import sympy as S
S.var('x:2 l',real=True)
J=S.Matrix([x0,x1]).norm()**2 + l*(1-x0-2*x1)
sol=S.solve(map(J.diff,[x0,x1,l]))
print(sol)
```

    {x0: 1/5, x1: 2/5, l: 2/5}

**Programming Tip.**

Using  the `Matrix` object is overkill for this problem but it
does demonstrate how Sympy's matrix machinery works. In this case,
we are using the `norm` method to compute the $L_2$  norm of the
given elements. Using `S.var` defines Sympy variables and injects
them into the global namespace. It is more Pythonic to do
something like `x0 = S.symbols('x0',real=True)` instead but the
other way is quicker, especially for variables with many
dimensions.

 The solution defines the exact point where the line is
tangent to the circle in [Figure](#fig:regularization_001).  The
Lagrange multiplier has incorporated the constraint into the objective
function.

```python
%matplotlib inline

from __future__ import division
import numpy as np
from numpy import pi, linspace, sqrt
from matplotlib.patches import Circle
from matplotlib.pylab import subplots

x1 = linspace(-1,1,10)
dx=linspace(.7,1.3,3)
fline = lambda x:(1-x)/2.

fig,ax=subplots()
_=ax.plot(dx*1/5,fline(dx*1/5),'s',ms=10,color='gray')
_=ax.plot(x1,fline(x1),color='gray',lw=3)
_=ax.add_patch(Circle((0,0),1/sqrt(5),alpha=0.3,color='gray'))
_=ax.plot(1/5,2/5,'o',color='k',ms=15)
_=ax.set_xlabel('$x_1$',fontsize=24)
_=ax.set_ylabel('$x_2$',fontsize=24)
_=ax.axis((-0.6,0.6,-0.6,0.6))
ax.set_aspect(1)

fig.tight_layout()
```

There is something subtle and very important about the nature of the solution,
however. Notice that there are other points very close to the solution on the
circle, indicated by the squares in [Figure](#fig:regularization_001). This
closeness could be a good thing, in case it helps us actually find a solution
in the first place, but it may be unhelpful in so far as it creates ambiguity.
Let's hold that thought and try the same problem using the $L_1$ norm instead
of the $L_2$ norm. Recall that

$$
\Vert \mathbf{x}\Vert_1 = \sum_{i=1}^d \vert x_i \vert
$$

 where $d$ is the dimension of the vector $\mathbf{x}$. Thus, we can
reformulate the same problem in the $L_1$  norm as in the following,

$$
\begin{aligned}
& \underset{\mathbf{x}}{\text{minimize}}
& & \Vert\mathbf{x}\Vert_1 \\
& \text{subject to:}
& & x_1 + 2 x_2 = 1
\end{aligned}
$$

  It turns out that this problem is somewhat harder to
solve using Sympy, but we have convex optimization modules in Python
that can help.

```python
from cvxpy import Variable, Problem, Minimize, norm1, norm2
x=Variable(2,1,name='x')
constr=[np.matrix([[1,2]])*x==1]
obj=Minimize(norm1(x))
p= Problem(obj,constr)
p.solve()
print(x.value)
```

**Programming Tip.**

The `cvxy` module provides a unified and accessible interface to the powerful
`cvxopt` convex optimization package, as well as other open-source solver
packages.

 As shown in [Figure](#fig:regularization_002), the constant-norm
contour in the $L_1$ norm is shaped like a diamond instead of a circle.
Furthermore, the solutions found in each case are different.  Geometrically,
this is because inflating the circular $L_2$ reaches out in all directions
whereas the $L_1$ ball creeps out along the principal axes.  This effect is
much more pronounced in higher dimensional spaces where $L_1$-balls get more
spikey  [^spikey].  Like the $L_2$ case, there are also neighboring points on
the constraint line, but notice that these are not close to the boundary of the
corresponding $L_1$ ball, as they were in the $L_2$ case.  This means that
these would be harder to confuse with the optimal solution because they
correspond to a substantially different $L_1$ ball.

[^spikey]: We discussed the geometry of high dimensional space
when we covered the curse of dimensionality in the
statistics chapter.

To double-check our earlier $L_2$ result, we can also use the
`cvxpy` module to find the $L_2$ solution as in the following
code,

```python
constr=[np.matrix([[1,2]])*x==1]
obj=Minimize(norm2(x)) #L2 norm
p= Problem(obj,constr)
p.solve()
print(x.value)
```

The only change to the code is the $L_2$ norm and we get
the same solution as before.

Let's see what happens in higher dimensions for both $L_2$ and
$L_1$ as we move from two dimensions to four dimensions.

```python
x=Variable(4,1,name='x')
constr=[np.matrix([[1,2,3,4]])*x==1]
obj=Minimize(norm1(x))
p= Problem(obj,constr)
p.solve()
print(x.value)
```

And also in the $L_2$ case with the following code,

```python
constr=[np.matrix([[1,2,3,4]])*x==1]
obj=Minimize(norm2(x))
p= Problem(obj,constr)
p.solve()
print(x.value)
```

Note that the $L_1$ solution has selected out only one
dimension for the solution, as the other components are
effectively zero. This is not so with the $L_2$ solution, which
has meaningful elements in multiple coordinates.  This is because
the $L_1$ problem has many pointy corners in the four dimensional
space that poke at the hyperplane that is defined by the
constraint. This essentially means the subsets (namely, the points
at the corners) are found as solutions because these touch the
hyperplane. This effect becomes more pronounced in higher
dimensions, which is the main benefit of using the $L_1$ norm
as we will see in the next section.

```python
from matplotlib.patches import Rectangle, RegularPolygon

r=RegularPolygon((0,0),4,1/2,pi/2,alpha=0.5,color='gray')
fig,ax=subplots()
dx = np.array([-0.1,0.1])
_=ax.plot(dx,fline(dx),'s',ms=10,color='gray')
_=ax.plot(x1,fline(x1),color='gray',lw=3)
_=ax.plot(0,1/2,'o',color='k',ms=15)
_=ax.add_patch(r)
_=ax.set_xlabel('$x_1$',fontsize=24)
_=ax.set_ylabel('$x_2$',fontsize=24)
_=ax.axis((-0.6,0.6,-0.6,0.6))
_=ax.set_aspect(1)
fig.tight_layout()
```

The diamond is the $L_1$ ball in two dimensions and the line is the
constraint. The point of intersection is the solution to the optimization
problem. Note that for $L_1$ optimization, the two nearby points on the
constraint (squares) do not touch the $L_1$ ball. Compare this with
[Figure](#fig:regularization_001).

## Ridge Regression

Now that we have a sense of the geometry of the situation, let's revisit
our classic linear regression probem. To recap, we want to solve the following
problem,

$$
\min_{\boldsymbol{\beta}\in \mathbb{R}^p} \Vert y -
\mathbf{X}\boldsymbol{\beta}\Vert
$$

 where $\mathbf{X}=\left[
\mathbf{x}_1,\mathbf{x}_2,\ldots,\mathbf{x}_p \right]$ and $\mathbf{x}_i\in
\mathbb{R}^n$. Furthermore, we assume that the $p$ column vectors are linearly
independent (i.e., $\texttt{rank}(\mathbf{X})=p$). Linear regression produces
the $\boldsymbol{\beta}$ that minimizes the mean squared error above.  In the
case where $p=n$, there is a unique solution to this problem. However, when
$p<n$, then there are infinitely many solutions.

To make this concrete, let's work this out using Sympy. First,
let's define an example $\mathbf{X}$ and $\mathbf{y}$ matrix,

```python
import sympy as S
from sympy import Matrix
X = Matrix([[1,2,3],
            [3,4,5]])
y = Matrix([[1,2]]).T
```

Now, we can define our coefficient vector $\boldsymbol{\beta}$
using the following code,

```python
b0,b1,b2=S.symbols('b:3',real=True)
beta = Matrix([[b0,b1,b2]]).T # transpose
```

Next, we define the objective function we are trying to minimize

```python
obj=(X*beta -y).norm(ord=2)**2
```

**Programming Tip.**

The Sympy `Matrix` class has useful methods like the `norm` function
used above to define the objective function. The `ord=2` means we want
to use the $L_2$ norm. The expression in parenthesis evaluates to a
`Matrix` object.

 Note that it is helpful to define real variables using
the keyword argument whenever applicable because it relieves
Sympy's internal machinery of dealing with complex numbers.
Finally, we can use calculus to solve this by setting the
derivatives of the objective function to zero.

```python
sol=S.solve([obj.diff(i) for i in beta])
beta.subs(sol)
```

Notice that the solution does not uniquely specify all the components
of the `beta` variable. This is a consequence of the $p<n$ nature of this
problem where $p=2$ and $n=3$.  While the the existence of this ambiguity does
not alter the solution,

```python
obj.subs(sol)
```

But it does change the length of the solution vector
`beta`,

```python
beta.subs(sol).norm(2)
```

If we want to minimize this length we can easily
use the same calculus as before,

```python
S.solve((beta.subs(sol).norm()**2).diff())
```

This provides the solution of minimum length
in the $L_2$ sense,

```python
betaL2=beta.subs(sol).subs(b2,S.Rational(1,6))
betaL2
```

But what is so special about solutions of minimum length? For machine
learning, driving the objective function to zero is symptomatic of overfitting
the data. Usually, at the zero bound, the machine learning method has
essentially memorized the training data, which is bad for generalization. Thus,
we can effectively stall this problem by defining a region for the solution
that is away from the zero-bound.

$$
\begin{aligned}
& \underset{\boldsymbol{\beta}}{\text{minimize}}
& & \Vert y - \mathbf{X}\boldsymbol{\beta}\Vert_2^2 \\
& \text{subject to:}
& & \Vert\boldsymbol{\beta}\Vert_2 < c
\end{aligned}
$$

 where $c$ is the tuning parameter. Using the same process as before,
we can re-write this as the following,

$$
\min_{\boldsymbol{\beta}\in\mathbb{R}^p}\Vert
y-\mathbf{X}\boldsymbol{\beta}\Vert_2^2 +\alpha\Vert\boldsymbol{\beta}\Vert_2^2
$$

 where $\alpha$ is the tuning parameter. These are the *penalized* or
Lagrange forms of these problems derived from the constrained versions. The
objective function is penalized by the $\Vert\boldsymbol{\beta}\Vert_2$ term.
For $L_2$ penalization, this is called *ridge* regression.    This is
implemented in Scikit-learn as `Ridge`. The following code sets this up for
our example,

```python
from sklearn.linear_model import Ridge
clf = Ridge(alpha=100.0,fit_intercept=False)
clf.fit(np.array(X).astype(float),np.array(y).astype(float))
```

Note that the `alpha` scales of the penalty for the
$\Vert\boldsymbol{\beta}\Vert_2$. We set the `fit_intercept=False` argument to
omit the extra offset term from our example. The corresponding solution is the
following,

```python
print(clf.coef_)
```

To double-check the solution, we can use some optimization tools from
Scipy and our previous Sympy analysis, as in the following,

```python
from scipy.optimize import minimize
f   = S.lambdify((b0,b1,b2),obj+beta.norm()**2*100.)
g   = lambda x:f(x[0],x[1],x[2])
out = minimize(g,[.1,.2,.3]) # initial guess
out.x
```

**Programming Tip.**

We had to define the additional `g` function from the lambda function we
created from the Sympy expression in `f` because the `minimize` function
expects a single object vector as input instead of a three separate arguments.

 which produces the same answer as the `Ridge` object. To
better understand the meaning of this result, we can re-compute the
mean squared error solution to this problem in one step using matrix
algebra instead of calculus,

```python
betaLS=X.T*(X*X.T).inv()*y
betaLS
```

Notice that this solves the posited problem exactly,

```python
X*betaLS-y
```

This means that the first term in the objective function
goes to zero,

$$
\Vert y-\mathbf{X}\boldsymbol{\beta}_{LS}\Vert=0
$$

 But, let's examine the $L_2$ length of this solution versus
the ridge regression solution,

```python
print(betaLS.norm().evalf(), np.linalg.norm(clf.coef_))
```

Thus, the ridge regression solution is shorter in the $L_2$
sense, but the first term in the objective function is not zero for
ridge regression,

```python
print((y-X*clf.coef_.T).norm()**2)
```

Ridge regression solution trades fitting error
($\Vert y-\mathbf{X} \boldsymbol{\beta}\Vert_2$) for solution
length ($\Vert\boldsymbol{\beta}\Vert_2$).

Let's see this in action with a familiar example from
[ch:stats:sec:nnreg](#ch:stats:sec:nnreg).  Consider
[Figure](#fig:regularization_003).
For this example, we created our usual chirp signal and attempted to
fit it with a high-dimensional polynomial, as we did in
the section [ch:ml:sec:cv](#ch:ml:sec:cv).  The lower panel is the same except
with ridge
regression. The shaded gray area is the space between the true signal
and the approximant in both cases. The horizontal hash marks indicate
the subset of $x_i$ values that each regressor was trained on.
Thus, the training set represents a non-uniform sample of the
underlying chirp waveform.  The top panel shows the usual polynomial
regression. Note that the regressor fits the given points extremely
well, but fails at the endpoint. The ridge regressor misses many of
the points in the middle, as indicated by the gray area, but does not
overshoot at the ends as much as the plain polynomial regression. This
is the basic trade-off for ridge regression. The Jupyter/IPython
notebook has the code for this graph, but the main steps
are shown in the following,

```python
# create chirp signal
xi = np.linspace(0,1,100)[:,None]
# sample chirp randomly
xin= np.sort(np.random.choice(xi.flatten(),20,replace=False))[:,None]
# create sampled waveform
y = cos(2*pi*(xin+xin**2))
# create full waveform for reference
yi = cos(2*pi*(xi+xi**2))

# create polynomial features
qfit = PolynomialFeatures(degree=8) # quadratic
Xq = qfit.fit_transform(xin)
# reformat input as polynomial
Xiq = qfit.fit_transform(xi)

lr=LinearRegression() # create linear model
lr.fit(Xq,y) # fit linear model

# create ridge regression model and fit
clf = Ridge(alpha=1e-9,fit_intercept=False)
clf.fit(Xq,y)
```

```python
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import numpy as np
from numpy import cos, pi

np.random.seed(1234567)
xi = np.linspace(0,1,100)[:,None]
xin = np.linspace(0,1,20)[:,None]
xin= np.sort(np.random.choice(xi.flatten(),20,replace=False))[:,None]
f0 = 1 # init frequency
BW = 2
y = cos(2*pi*(f0*xin+(BW/2.0)*xin**2))
yi = cos(2*pi*(f0*xi+(BW/2.0)*xi**2))

qfit = PolynomialFeatures(degree=8) # quadratic
Xq = qfit.fit_transform(xin)
Xiq = qfit.fit_transform(xi)

lr=LinearRegression() # create linear model
_=lr.fit(Xq,y)

fig,axs=subplots(2,1,sharex=True,sharey=True)
fig.set_size_inches((6,6))
ax=axs[0]
_=ax.plot(xi,yi,label='true',ls='--',color='k')
_=ax.plot(xi,lr.predict(Xiq),label=r'$\beta_{LS}$',color='k')
_=ax.legend(loc=0)
_=ax.set_ylabel(r'$\hat{y}$  ',fontsize=22,rotation='horizontal')
_=ax.fill_between(xi.flatten(),yi.flatten(),lr.predict(Xiq).flatten(),color='gray',alpha=.3)
_=ax.set_title('Polynomial Regression of Chirp Signal')
_=ax.plot(xin, -1.5+np.array([0.01]*len(xin)), '|', color='k',mew=3)

clf = Ridge(alpha=1e-9,fit_intercept=False)
_=clf.fit(Xq,y)
ax=axs[1]
_=ax.plot(xi,yi,label=r'true',ls='--',color='k')
_=ax.plot(xi,clf.predict(Xiq),label=r'$\beta_{RR}$',color='k')
_=ax.legend(loc=(0.25,0.70))
_=ax.fill_between(xi.flatten(),yi.flatten(),clf.predict(Xiq).flatten(),color='gray',alpha=.3)
# add rug plot
_=ax.plot(xin, -1.5+np.array([0.01]*len(xin)), '|', color='k',mew=3)
_=ax.set_xlabel('$x$',fontsize=22)
_=ax.set_ylabel(r'$\hat{y}$  ',fontsize=22,rotation='horizontal')
_=ax.set_title('Ridge Regression of Chirp Signal')
```

The top figure shows polynomial regression and the lower panel shows
polynomial ridge regression. The ridge regression does not match as well
throughout most of the domain, but it does not flare as violently at the ends.
This is because the ridge constraint holds the coefficient vector down at the
expense of poorer performance along the middle of the domain.

## Lasso

Lasso regression follows the same basic pattern as ridge regression,
except with the $L_1$ norm in the objective function.

$$
\min_{\boldsymbol{\beta}\in\mathbb{R}^p}\Vert
y-\mathbf{X}\boldsymbol{\beta}\Vert^2 +\alpha\Vert\boldsymbol{\beta}\Vert_1
$$

 The interface in Scikit-learn is likewise the same.
The following is the same problem as before using lasso
instead of ridge regression,

```python
X = np.matrix([[1,2,3],
               [3,4,5]])
y = np.matrix([[1,2]]).T
from sklearn.linear_model import Lasso
lr = Lasso(alpha=1.0,fit_intercept=False)
_=lr.fit(X,y)
print(lr.coef_)
```

As before, we can use the optimization tools in Scipy to solve this
also,

```python
from scipy.optimize import fmin
obj = 1/4.*(X*beta-y).norm(2)**2 + beta.norm(1)*l
f = S.lambdify((b0,b1,b2),obj.subs(l,1.0))
g = lambda x:f(x[0],x[1],x[2])
fmin(g,[0.1,0.2,0.3])
```

**Programming Tip.**

The `fmin` function from Scipy's optimization module uses an
algorithm that does not depend upon derivatives. This is useful
because, unlike the $L_2$ norm, the $L_1$ norm has sharp corners
that make it harder to estimate derivatives.

 This result matches the previous one from the
Scikit-learn `Lasso` object. Solving it using Scipy is motivating
and provides a good sanity check, but specialized algorithms are
required in practice. The following code block re-runs the lasso
with varying $\alpha$ and plots the coefficients in
[Figure](#fig:regularization_004). Notice that as $\alpha$ increases, all
but one of the coefficients is driven to zero. Increasing $\alpha$
makes the trade-off between fitting the data in the $L_2$ sense
and wanting to reduce the number of nonzero coefficients
(equivalently, the number of features used) in the model. For a
given problem, it may be more practical to focus on reducing the
number of features in the model (i.e., large $\alpha$) than the
quality of the data fit in the training data. The lasso provides a
clean way to navigate this trade-off.

The following code loops over a set of $\alpha$ values and
collects the corresponding lasso coefficients to be plotted
in [Figure](#fig:regularization_004)

```python
o=[]
alphas= np.logspace(-3,0,10)
for a in alphas:
    clf = Lasso(alpha=a,fit_intercept=False)
    _=clf.fit(X,y)
    o.append(clf.coef_)
```

```python
fig,ax=subplots()
fig.set_size_inches((8,5))
k=np.vstack(o)
ls = ['-','--',':','-.']
for i in range(k.shape[1]):
    _=ax.semilogx(alphas,k[:,i],'o-',
                label='coef %d'%(i),
                color='k',ls=ls[i],
                alpha=.8,)

_=ax.axis(ymin=-1e-1)
_=ax.legend(loc=0)
_=ax.set_xlabel(r'$\alpha$',fontsize=20)
_=ax.set_ylabel(r'Lasso coefficients',fontsize=16)
fig.tight_layout()
```

As $\alpha$ increases, more of the model coefficients are driven to zero for
lasso regression.


# Table of Contents
* [1b. Fixed flux spinodal decomposition on a square domain](#1b.-Fixed-flux-spinodal-decomposition-on-a-square-domain)
	* [Use Binder For Live Examples](#Use-Binder-For-Live-Examples)
	* [Define $f_0$](#Define-$f_0$)
	* [Define the Equation](#Define-the-Equation)
	* [Solve the Equation](#Solve-the-Equation)
	* [Run the Example Locally](#Run-the-Example-Locally)
	* [Movie of Evolution](#Movie-of-Evolution)

# 1b. Fixed flux spinodal decomposition on a square domain

## Use Binder For Live Examples

[](http://mybinder.org/repo/wd15/fipy-hackathon1)

The free energy is given by,

$$ f_0\left[ c \left( \vec{r} \right) \right] =
   - \frac{A}{2} \left(c - c_m\right)^2
   + \frac{B}{4} \left(c - c_m\right)^4
   + \frac{c_{\alpha}}{4} \left(c - c_{\alpha} \right)^4
   + \frac{c_{\beta}}{4} \left(c - c_{\beta} \right)^4 $$

In FiPy we write the evolution equation as

$$ \frac{\partial c}{\partial t} = \nabla \cdot  \left[
       D \left( c \right) \left( \frac{ \partial^2 f_0 }{ \partial c^2} \nabla c - \kappa \nabla \nabla^2 c \right)
   \right] $$

Let's start by calculating $ \frac{ \partial^2 f_0 }{ \partial c^2} $ using sympy. It's easy for this case, but useful in the general case for taking care of difficult book keeping in phase field problems.

```python
%matplotlib inline

import sympy
import fipy as fp
import numpy as np
```

```python
A, c, c_m, B, c_alpha, c_beta = sympy.symbols("A c_var c_m B c_alpha c_beta")
```

```python
f_0 = - A / 2 * (c - c_m)**2 + B / 4 * (c - c_m)**4 + c_alpha / 4 * (c - c_alpha)**4 + c_beta / 4 * (c - c_beta)**4
```

```python
print f_0
```

    -A*(-c_m + c_var)**2/2 + B*(-c_m + c_var)**4/4 + c_alpha*(-c_alpha + c_var)**4/4 + c_beta*(-c_beta + c_var)**4/4

```python
sympy.diff(f_0, c, 2)
```

    -A + 3*B*(c_m - c_var)**2 + 3*c_alpha*(c_alpha - c_var)**2 + 3*c_beta*(c_beta - c_var)**2

The first step in implementing any problem in FiPy is to define the mesh. For [Problem 1a]({{ site.baseurl }}/hackathon1/problems.ipynb/#a.-Square-Periodic) the solution domain is just a square domain, but the boundary conditions are periodic, so a `PeriodicGrid2D` object is used. No other boundary conditions are required.

```python
mesh = fp.Grid2D(nx=50, ny=50, dx=0.5, dy=0.5)
```

The next step is to define the parameters and create a solution variable.

```python
c_alpha = 0.05
c_beta = 0.95
A = 2.0
kappa = 2.0
c_m = (c_alpha + c_beta) / 2.
B = A / (c_alpha - c_m)**2
D = D_alpha = D_beta = 2. / (c_beta - c_alpha)
c_0 = 0.45
q = np.sqrt((2., 3.))
epsilon = 0.01

c_var = fp.CellVariable(mesh=mesh, name=r"$c$", hasOld=True)
```

Now we need to define the initial conditions given by,

Set $c\left(\vec{r}, t\right)$ such that

$$ c\left(\vec{r}, 0\right) = \bar{c}_0 + \epsilon \cos \left( \vec{q} \cdot \vec{r} \right) $$

```python
r = np.array((mesh.x, mesh.y))
c_var[:] = c_0 + epsilon * np.cos((q[:, None] * r).sum(0))
viewer = fp.Viewer(c_var)
```

## Define $f_0$

To define the equation with FiPy first define `f_0` in terms of FiPy. Recall `f_0` from above calculated using Sympy. Here we use the string representation and set it equal to `f_0_var` using the `exec` command.

```python
out = sympy.diff(f_0, c, 2)
```

```python
exec "f_0_var = " + repr(out)
```

```python
#f_0_var = -A + 3*B*(c_var - c_m)**2 + 3*c_alpha*(c_var - c_alpha)**2 + 3*c_beta*(c_var - c_beta)**2
f_0_var
```

    (((((pow((0.5 - $c$), 2)) * 29.6296296296) + -2.0) + ((pow((0.05 - $c$), 2)) * 0.15)) + ((pow((0.95 - $c$), 2)) * 2.85))

## Define the Equation

```python
eqn = fp.TransientTerm(coeff=1.) == fp.DiffusionTerm(D * f_0_var) - fp.DiffusionTerm((D, kappa))
eqn
```

    (TransientTerm(coeff=1.0) + (DiffusionTerm(coeff=[-(((((((pow((0.5 - $c$), 2)) * 29.6296296296) + -2.0) + ((pow((0.05 - $c$), 2)) * 0.15)) + ((pow((0.95 - $c$), 2)) * 2.85)) * 2.22222222222))]) + DiffusionTerm(coeff=[2.2222222222222223, 2.0])))

## Solve the Equation

To solve the equation a simple time stepping scheme is used which is decreased or increased based on whether the residual decreases or increases. A time step is recalculated if the required tolerance is not reached.

```python
elapsed = 0.0
steps = 0
dt = 0.01
total_sweeps = 2
tolerance = 1e-1
total_steps = 100
```

```python
c_var[:] = c_0 + epsilon * np.cos((q[:, None] * r).sum(0))
c_var.updateOld()
from fipy.solvers.pysparse import LinearLUSolver as Solver
solver = Solver()

while steps < total_steps:
    res0 = eqn.sweep(c_var, dt=dt, solver=solver)

    for sweeps in range(total_sweeps):
        res = eqn.sweep(c_var, dt=dt, solver=solver)

    if res < res0 * tolerance:
        steps += 1
        elapsed += dt
        dt *= 1.1
        c_var.updateOld()
    else:
        dt *= 0.8
        c_var[:] = c_var.old

viewer.plot()
print 'elapsed_time:',elapsed
```

## Run the Example Locally

The following cell will dumpy a file called `fipy_hackathon1b.py` to the local file system to be run. The images are saved out at each time step.

```python
%%writefile fipy_hackathon_1b.py

import fipy as fp
import numpy as np

mesh = fp.Grid2D(nx=400, ny=400, dx=0.5, dy=0.5)

c_alpha = 0.05
c_beta = 0.95
A = 2.0
kappa = 2.0
c_m = (c_alpha + c_beta) / 2.
B = A / (c_alpha - c_m)**2
D = D_alpha = D_beta = 2. / (c_beta - c_alpha)
c_0 = 0.45
q = np.sqrt((2., 3.))
epsilon = 0.01

c_var = fp.CellVariable(mesh=mesh, name=r"$c$", hasOld=True)

r = np.array((mesh.x, mesh.y))
c_var[:] = c_0 + epsilon * np.cos((q[:, None] * r).sum(0))

f_0_var = -A + 3*B*(c_var - c_m)**2 + 3*c_alpha*(c_var - c_alpha)**2 + 3*c_beta*(c_var - c_beta)**2

eqn = fp.TransientTerm(coeff=1.) == fp.DiffusionTerm(D * f_0_var) - fp.DiffusionTerm((D, kappa))

elapsed = 0.0
steps = 0
dt = 0.01
total_sweeps = 2
tolerance = 1e-1
total_steps = 300

c_var[:] = c_0 + epsilon * np.cos((q[:, None] * r).sum(0))

c_var.updateOld()

from fipy.solvers.pysparse import LinearLUSolver as Solver

solver = Solver()

viewer = fp.Viewer(c_var)
while steps < total_steps:
    res0 = eqn.sweep(c_var, dt=dt, solver=solver)

    for sweeps in range(total_sweeps):
        res = eqn.sweep(c_var, dt=dt, solver=solver)

        print ' '
        print 'steps',steps
        print 'res',res
        print 'sweeps',sweeps
        print 'dt',dt

    if res < res0 * tolerance:
        steps += 1
        elapsed += dt
        dt *= 1.1
        if steps % 1 == 0:
             viewer.plot('image{0}.png'.format(steps))
        c_var.updateOld()
    else:
        dt *= 0.8
        c_var[:] = c_var.old
```

    Writing fipy_hackathon_1b.py

## Movie of Evolution

The movie of the evolution for 300 steps.

The movie was generated with the output files of the form `image*.png` using the following commands,

    $ rename 's/\d+/sprintf("%05d",$&)/e' image*
    $ ffmpeg -f image2 -r 6 -i 'image%05d.png' output.mp4

```python
from IPython.display import YouTubeVideo
scale = 1.5
YouTubeVideo('S-EYiO0EltM', width=420 * scale, height=315 * scale, rel=0)
```

```python

```


# Detrending, Stylized Facts and the Business Cycle

In an influential article, Harvey and Jaeger (1993) described the use of unobserved components models (also known as "structural time series models") to derive stylized facts of the business cycle.

Their paper begins:

    "Establishing the 'stylized facts' associated with a set of time series is widely considered a crucial step
    in macroeconomic research ... For such facts to be useful they should (1) be consistent with the stochastic
    properties of the data and (2) present meaningful information."

In particular, they make the argument that these goals are often better met using the unobserved components approach rather than the popular Hodrick-Prescott filter or Box-Jenkins ARIMA modeling techniques.

statsmodels has the ability to perform all three types of analysis, and below we follow the steps of their paper, using a slightly updated dataset.

```python
%matplotlib inline

import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt

from IPython.display import display, Latex
```

## Unobserved Components

The unobserved components model available in statsmodels can be written as:

$$
y_t = \underbrace{\mu_{t}}_{\text{trend}} + \underbrace{\gamma_{t}}_{\text{seasonal}} + \underbrace{c_{t}}_{\text{cycle}} + \sum_{j=1}^k \underbrace{\beta_j x_{jt}}_{\text{explanatory}} + \underbrace{\varepsilon_t}_{\text{irregular}}
$$

see Durbin and Koopman 2012, Chapter 3 for notation and additional details. Notice that different specifications for the different individual components can support a wide range of models. The specific models considered in the paper and below are specializations of this general equation.

### Trend

The trend component is a dynamic extension of a regression model that includes an intercept and linear time-trend.

$$
\begin{align}
\underbrace{\mu_{t+1}}_{\text{level}} & = \mu_t + \nu_t + \eta_{t+1} \qquad & \eta_{t+1} \sim N(0, \sigma_\eta^2) \\\\
\underbrace{\nu_{t+1}}_{\text{trend}} & = \nu_t + \zeta_{t+1} & \zeta_{t+1} \sim N(0, \sigma_\zeta^2) \\
\end{align}
$$

where the level is a generalization of the intercept term that can dynamically vary across time, and the trend is a generalization of the time-trend such that the slope can dynamically vary across time.

For both elements (level and trend), we can consider models in which:

- The element is included vs excluded (if the trend is included, there must also be a level included).
- The element is deterministic vs stochastic (i.e. whether or not the variance on the error term is confined to be zero or not)

The only additional parameters to be estimated via MLE are the variances of any included stochastic components.

This leads to the following specifications:

|                                                                      | Level | Trend | Stochastic Level | Stochastic Trend |
|----------------------------------------------------------------------|-------|-------|------------------|------------------|
| Constant                                                             | ✓     |       |                  |                  |
| Local Level
 (random walk)                                     | ✓     |       | ✓                |                  |
| Deterministic trend                                                  | ✓     | ✓     |                  |                  |
| Local level with deterministic trend
 (random walk with drift) | ✓     | ✓     | ✓                |                  |
| Local linear trend                                                   | ✓     | ✓     | ✓                | ✓                |
| Smooth trend
 (integrated random walk)                         | ✓     | ✓     |                  | ✓                |

### Seasonal

The seasonal component is written as:

$$
\gamma_t = - \sum_{j=1}^{s-1} \gamma_{t+1-j} + \omega_t \qquad \omega_t \sim N(0, \sigma_\omega^2)
$$

The periodicity (number of seasons) is `s`, and the defining character is that (without the error term), the seasonal components sum to zero across one complete cycle. The inclusion of an error term allows the seasonal effects to vary over time.

The variants of this model are:

- The periodicity `s`
- Whether or not to make the seasonal effects stochastic.

If the seasonal effect is stochastic, then there is one additional parameter to estimate via MLE (the variance of the error term).

### Cycle

The cyclical component is intended to capture cyclical effects at time frames much longer than captured by the seasonal component. For example, in economics the cyclical term is often intended to capture the business cycle, and is then expected to have a period between "1.5 and 12 years" (see Durbin and Koopman).

The cycle is written as:

$$
\begin{align}
c_{t+1} & = c_t \cos \lambda_c + c_t^* \sin \lambda_c + \tilde \omega_t \qquad & \tilde \omega_t \sim N(0, \sigma_{\tilde \omega}^2) \\\\
c_{t+1}^* & = -c_t \sin \lambda_c + c_t^* \cos \lambda_c + \tilde \omega_t^* & \tilde \omega_t^* \sim N(0, \sigma_{\tilde \omega}^2)
\end{align}
$$

The parameter $\lambda_c$ (the frequency of the cycle) is an additional parameter to be estimated by MLE. If the seasonal effect is stochastic, then there is one another parameter to estimate (the variance of the error term - note that both of the error terms here share the same variance, but are assumed to have independent draws).

### Irregular

The irregular component is assumed to be a white noise error term. Its variance is a parameter to be estimated by MLE; i.e.

$$
\varepsilon_t \sim N(0, \sigma_\varepsilon^2)
$$

In some cases, we may want to generalize the irregular component to allow for autoregressive effects:

$$
\varepsilon_t = \rho(L) \varepsilon_{t-1} + \epsilon_t, \qquad \epsilon_t \sim N(0, \sigma_\epsilon^2)
$$

In this case, the autoregressive parameters would also be estimated via MLE.

### Regression effects

We may want to allow for explanatory variables by including additional terms

$$
\sum_{j=1}^k \beta_j x_{jt}
$$

or for intervention effects by including

$$
\begin{align}
\delta w_t \qquad \text{where} \qquad w_t & = 0, \qquad t < \tau, \\\\
& = 1, \qquad t \ge \tau
\end{align}
$$

These additional parameters could be estimated via MLE or by including them as components of the state space formulation.

## Data

Following Harvey and Jaeger, we will consider the following time series:

- US real GNP, "output", ([GNPC96](https://research.stlouisfed.org/fred2/series/GNPC96))
- US GNP implicit price deflator, "prices", ([GNPDEF](https://research.stlouisfed.org/fred2/series/GNPDEF))
- US monetary base, "money", ([AMBSL](https://research.stlouisfed.org/fred2/series/AMBSL))

The time frame in the original paper varied across series, but was broadly 1954-1989. Below we use data from the period 1948-2008 for all series. Although the unobserved components approach allows isolating a seasonal component within the model, the series considered in the paper, and here, are already seasonally adjusted.

All data series considered here are taken from [Federal Reserve Economic Data (FRED)](https://research.stlouisfed.org/fred2/). Conveniently, the Python library [Pandas](https://pandas.pydata.org/) has the ability to download data from FRED directly.

```python
# Datasets
from pandas_datareader.data import DataReader

# Get the raw data
start = '1948-01'
end = '2008-01'
us_gnp = DataReader('GNPC96', 'fred', start=start, end=end)
us_gnp_deflator = DataReader('GNPDEF', 'fred', start=start, end=end)
us_monetary_base = DataReader('AMBSL', 'fred', start=start, end=end).resample('QS').mean()
recessions = DataReader('USRECQ', 'fred', start=start, end=end).resample('QS').last().values[:,0]

# Construct the dataframe
dta = pd.concat(map(np.log, (us_gnp, us_gnp_deflator, us_monetary_base)), axis=1)
dta.columns = ['US GNP','US Prices','US monetary base']
dates = dta.index._mpl_repr()
```

To get a sense of these three variables over the timeframe, we can plot them:

```python
# Plot the data
ax = dta.plot(figsize=(13,3))
ylim = ax.get_ylim()
ax.xaxis.grid()
ax.fill_between(dates, ylim[0]+1e-5, ylim[1]-1e-5, recessions, facecolor='k', alpha=0.1);
```

## Model

Since the data is already seasonally adjusted and there are no obvious explanatory variables, the generic model considered is:

$$
y_t = \underbrace{\mu_{t}}_{\text{trend}} + \underbrace{c_{t}}_{\text{cycle}} + \underbrace{\varepsilon_t}_{\text{irregular}}
$$

The irregular will be assumed to be white noise, and the cycle will be stochastic and damped. The final modeling choice is the specification to use for the trend component. Harvey and Jaeger consider two models:

1. Local linear trend (the "unrestricted" model)
2. Smooth trend (the "restricted" model, since we are forcing $\sigma_\eta = 0$)

Below, we construct `kwargs` dictionaries for each of these model types. Notice that rather that there are two ways to specify the models. One way is to specify components directly, as in the table above. The other way is to use string names which map to various specifications.

```python
# Model specifications

# Unrestricted model, using string specification
unrestricted_model = {
    'level': 'local linear trend', 'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
}

# Unrestricted model, setting components directly
# This is an equivalent, but less convenient, way to specify a
# local linear trend model with a stochastic damped cycle:
# unrestricted_model = {
#     'irregular': True, 'level': True, 'stochastic_level': True, 'trend': True, 'stochastic_trend': True,
#     'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
# }

# The restricted model forces a smooth trend
restricted_model = {
    'level': 'smooth trend', 'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
}

# Restricted model, setting components directly
# This is an equivalent, but less convenient, way to specify a
# smooth trend model with a stochastic damped cycle. Notice
# that the difference from the local linear trend model is that
# `stochastic_level=False` here.
# unrestricted_model = {
#     'irregular': True, 'level': True, 'stochastic_level': False, 'trend': True, 'stochastic_trend': True,
#     'cycle': True, 'damped_cycle': True, 'stochastic_cycle': True
# }
```

We now fit the following models:

1. Output, unrestricted model
2. Prices, unrestricted model
3. Prices, restricted model
4. Money, unrestricted model
5. Money, restricted model

```python
# Output
output_mod = sm.tsa.UnobservedComponents(dta['US GNP'], **unrestricted_model)
output_res = output_mod.fit(method='powell', disp=False)

# Prices
prices_mod = sm.tsa.UnobservedComponents(dta['US Prices'], **unrestricted_model)
prices_res = prices_mod.fit(method='powell', disp=False)

prices_restricted_mod = sm.tsa.UnobservedComponents(dta['US Prices'], **restricted_model)
prices_restricted_res = prices_restricted_mod.fit(method='powell', disp=False)

# Money
money_mod = sm.tsa.UnobservedComponents(dta['US monetary base'], **unrestricted_model)
money_res = money_mod.fit(method='powell', disp=False)

money_restricted_mod = sm.tsa.UnobservedComponents(dta['US monetary base'], **restricted_model)
money_restricted_res = money_restricted_mod.fit(method='powell', disp=False)
```

Once we have fit these models, there are a variety of ways to display the information. Looking at the model of US GNP, we can summarize the fit of the model using the `summary` method on the fit object.

```python
print(output_res.summary())
```

                                Unobserved Components Results
    =====================================================================================
    Dep. Variable:                        US GNP   No. Observations:                  241
    Model:                    local linear trend   Log Likelihood                 769.632
                       + damped stochastic cycle   AIC                          -1527.264
    Date:                       Tue, 24 Dec 2019   BIC                          -1506.455
    Time:                               15:05:42   HQIC                         -1518.876
    Sample:                           01-01-1948
                                    - 01-01-2008
    Covariance Type:                         opg
    ====================================================================================
                           coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------------
    sigma2.irregular  1.091e-06    7.3e-06      0.150      0.881   -1.32e-05    1.54e-05
    sigma2.level      4.078e-06   4.94e-05      0.083      0.934   -9.27e-05       0.000
    sigma2.trend      2.963e-06    1.4e-06      2.112      0.035    2.14e-07    5.71e-06
    sigma2.cycle      3.911e-05   2.57e-05      1.524      0.128   -1.12e-05    8.94e-05
    frequency.cycle      0.4454      0.047      9.474      0.000       0.353       0.537
    damping.cycle        0.8684      0.042     20.726      0.000       0.786       0.951
    ===================================================================================
    Ljung-Box (Q):                       43.93   Jarque-Bera (JB):                 9.29
    Prob(Q):                              0.31   Prob(JB):                         0.01
    Heteroskedasticity (H):               0.27   Skew:                            -0.05
    Prob(H) (two-sided):                  0.00   Kurtosis:                         3.96
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using the outer product of gradients (complex-step).

For unobserved components models, and in particular when exploring stylized facts in line with point (2) from the introduction, it is often more instructive to plot the estimated unobserved components (e.g. the level, trend, and cycle) themselves to see if they provide a meaningful description of the data.

The `plot_components` method of the fit object can be used to show plots and confidence intervals of each of the estimated states, as well as a plot of the observed data versus the one-step-ahead predictions of the model to assess fit.

```python
fig = output_res.plot_components(legend_loc='lower right', figsize=(15, 9));
```

Finally, Harvey and Jaeger summarize the models in another way to highlight the relative importances of the trend and cyclical components; below we replicate their Table I. The values we find are broadly consistent with, but different in the particulars from, the values from their table.

```python
# Create Table I
table_i = np.zeros((5,6))

start = dta.index[0]
end = dta.index[-1]
time_range = '%d:%d-%d:%d' % (start.year, start.quarter, end.year, end.quarter)
models = [
    ('US GNP', time_range, 'None'),
    ('US Prices', time_range, 'None'),
    ('US Prices', time_range, r'$\sigma_\eta^2 = 0$'),
    ('US monetary base', time_range, 'None'),
    ('US monetary base', time_range, r'$\sigma_\eta^2 = 0$'),
]
index = pd.MultiIndex.from_tuples(models, names=['Series', 'Time range', 'Restrictions'])
parameter_symbols = [
    r'$\sigma_\zeta^2$', r'$\sigma_\eta^2$', r'$\sigma_\kappa^2$', r'$\rho$',
    r'$2 \pi / \lambda_c$', r'$\sigma_\varepsilon^2$',
]

i = 0
for res in (output_res, prices_res, prices_restricted_res, money_res, money_restricted_res):
    if res.model.stochastic_level:
        (sigma_irregular, sigma_level, sigma_trend,
         sigma_cycle, frequency_cycle, damping_cycle) = res.params
    else:
        (sigma_irregular, sigma_level,
         sigma_cycle, frequency_cycle, damping_cycle) = res.params
        sigma_trend = np.nan
    period_cycle = 2 * np.pi / frequency_cycle

    table_i[i, :] = [
        sigma_level*1e7, sigma_trend*1e7,
        sigma_cycle*1e7, damping_cycle, period_cycle,
        sigma_irregular*1e7
    ]
    i += 1

pd.set_option('float_format', lambda x: '%.4g' % np.round(x, 2) if not np.isnan(x) else '-')
table_i = pd.DataFrame(table_i, index=index, columns=parameter_symbols)
table_i
```

      $\sigma_\zeta^2$
      $\sigma_\eta^2$
      $\sigma_\kappa^2$
      $\rho$
      $2 \pi / \lambda_c$
      $\sigma_\varepsilon^2$

      Series
      Time range
      Restrictions

      US GNP
      1948:1-2008:1
      None
      40.78
      29.63
      391.1
      0.87
      14.11
      10.91

      US Prices
      1948:1-2008:1
      None
      26.53
      31.8
      0.11
      0.92
      9.93
      16.02

      $\sigma_\eta^2 = 0$
      22.02
      -
      49.25
      0.89
      15.35
      5.02

      US monetary base
      1948:1-2008:1
      None
      63.45
      19
      196.3
      0.89
      23.15
      0

      $\sigma_\eta^2 = 0$
      18.91
      -
      247.3
      0.89
      23.8
      0


## Chapter 10

```python
import sympy
import numpy as np
sympy.init_printing(use_latex='mathjax')
s, w = sympy.symbols('s w', real=True)
import control
import matplotlib.pyplot as plt
from scipy.optimize import minimize
```

### Problem 10.1
####  a)
##### i)
The spectral density $S(\omega)$ is:
\begin{eqnarray}
S(\omega) &=& H(-j \omega) H^T(j \omega) \\
&=& \frac{1}{(\omega^2 + \alpha^2)(\omega^2 + \beta^2)} \\
&=& \frac{1}{\beta^2 - \alpha^2} \biggl [ \frac{1}{\omega^2 + \alpha^2} - \frac{1}{\omega^2 + \beta^2} \biggr ]
\end{eqnarray}
where the last result comes from partial fraction decomposition.

##### ii)
\begin{eqnarray}
\lim_{T \to \infty} \int_{-T/2}^{T/2} x^2 (\tau) d \tau &=& \rho(0) \\
&\triangleq& \frac{1}{2 \pi} \int_{-\infty}^\infty S(\omega) d \omega \\
&=& \frac{1}{2 \pi} \frac{1}{\beta^2 - \alpha^2} \int_{-\infty}^\infty \biggl [ \frac{1}{\omega^2 + \alpha^2} - \frac{1}{\omega^2 + \beta^2} \biggr ] \\
&=& \frac{1}{2 \pi} \frac{1}{\beta^2 - \alpha^2} \biggl [ \frac{\pi}{\alpha} - \frac{\pi}{\beta} \biggr ] \\
&=& \frac{1}{2} \frac{1}{(\beta + \alpha)(\beta - \alpha)} \biggl [ \frac{\beta - \alpha}{\alpha \beta} \biggr ] \\
&=& \frac{1}{2} \frac{1}{\alpha \beta(\beta + \alpha)}
\end{eqnarray}

##### iii)
Put the transfer function into the "first-companion" form:

```python
p1, p2, p3, a, b = sympy.symbols('p1 p2 p3 a b')
# use first companion form (Eqn. 3.88)
A = sympy.Matrix([[0, 1], [-a*b, -(a+b)]])
F = sympy.Matrix([[0], [1]])
Q = 1
P = sympy.Matrix([[p1, p2], [p2, p3]])
Z = A*P + P*A.T + F*Q*F.T
# Z == [[0, 0], [0, 0]], so 3 equations and 3 unknowns
sympy.pprint('{} = 0'.format(Z[0, 0]))
sympy.pprint('{} = 0'.format(Z[0, 1]))
sympy.pprint('{} = 0'.format(Z[1, 1]))
```

    2*p2 = 0
    -a*b*p1 + p2*(-a - b) + p3 = 0
    -2*a*b*p2 + 2*p3*(-a - b) + 1 = 0

The above linear equation leads to the covariance matrix:
\begin{eqnarray}
\bar P = \begin{pmatrix}
\frac{1}{2 \alpha \beta (\alpha+\beta)} & 0 \\
0 & \frac{1}{2 \alpha \beta}
\end{pmatrix},
\end{eqnarray}
and, for the mean-squared value, $\rho(0) = C^T \bar P e^{A \cdot 0} C = C^T \bar P C$.

```python
P = sympy.Matrix([[1/(2*a*b*(a+b)), 0], [0, 1/(2*a*b)]])
C = sympy.Matrix([[1], [0]])  # from the first companion form definition
qdrtc = sympy.simplify((C.T*P*C)[0])
sympy.pprint(qdrtc)
```

          1
    ─────────────
    2⋅a⋅b⋅(a + b)

####  b)
_omitted because it's basically the same problem as above_

### Problem 10.2
_this is how I would solve the problem, but the kernel keeps crashing so I'm moving on_

```python
# data to use for regression
frequencies = np.array([0.1, 0.35, 1., 5., 100.])
Sw = np.array([90., 19000., 10., 1.])
# objective function - solve an optimization problem for the coefficients in feature
def obj(feature, freqs, powSpecDens):
    K, a, icsi1, icsi2, icsi3, w1, w2, w3 = feature
    def S(w):
        num = K**2 * w**4 * ( (w3-w)**2 + 4.*icsi3**2*w3**2*w**2 )
        den = ( (w**2 + a**2) * ( (w1-w)**2 + 4.*icsi1**2*w1**2*w**2 )*( (w2-w)**2 + 4.*icsi2**2*w2**2*w**2 ) )
        return num / den
    return np.sum([(S(f) - psd)**2 for f,psd in zip(frequencies, powSpecDens)])
# initial guess
feature = np.array([100., 1., 100., 100., 100., 0.5, 0.5, 0.5])
# bounds
bnds = ((0, 1000),(0, 10),(0, 1000),(0, 1000),(0, 1000),(0, 1),(0, 1),(0, 1))
# solve optimization problem
sol = minimize(obj, feature, args=(frequencies, Sw), bounds=bnds)
print(sol)
```

### Problem 10.3
#### a)
The covariance equation can be written as:
\begin{eqnarray}
\dot P = AP + PA^T + FQ_vF^T,
\end{eqnarray}
where $Q_v$ is the diagonal matrix representing the expected value of the white-noise and
\begin{eqnarray}
A = \begin{pmatrix}
0 & 1 & 0 \\
0 & 0 & -g \\
0 & \frac{1}{R} & 0
\end{pmatrix}, F = \begin{pmatrix}
0 & 0 \\
1 & 0 \\
0 & 1
\end{pmatrix}.
\end{eqnarray}

#### b)
_There are no numbers for the Q matrix, so just setup the problem_

```python
# numbers from Example 3E
Omega = 1.235e-3
g = 9.8
R = g / Omega**2
A = np.array([[0, 1, 0], [0, 0, -g], [0, 1./R, 0]]).astype('float')
F = np.array([[0, 0], [1, 0], [0, 1]]).astype('float')
p1d, p2d, p3d, p4d, p5d, p6d = sympy.symbols('p1d p2d p3d p4d p5d p6d')
p1, p2, p3, p4, p5, p6 = sympy.symbols('p1 p2 p3 p4 p5 p6')
lhs = sympy.Matrix([[p1d, p2d, p3d], [p2d, p4d, p5d], [p3d, p5d, p6d]])
P = sympy.Matrix([[p1, p2, p3], [p2, p4, p5], [p3, p5, p6]])
w1, w2 = sympy.symbols('w1 w2')
Q = sympy.Matrix([[w1, 0], [0, w2]])
rhs = sympy.Matrix(A)*P + P*sympy.Matrix(A.T) + sympy.Matrix(F)*Q*sympy.Matrix(F.transpose())
print('{} = {}'.format(lhs, rhs))
```

    Matrix([[p1d, p2d, p3d], [p2d, p4d, p5d], [p3d, p5d, p6d]]) = Matrix([[2.0*p2, -9.8*p3 + 1.0*p4, 1.55635204081633e-7*p2 + 1.0*p5], [-9.8*p3 + 1.0*p4, -19.6*p5 + 1.0*w1, 1.55635204081633e-7*p4 - 9.8*p6], [1.55635204081633e-7*p2 + 1.0*p5, 1.55635204081633e-7*p4 - 9.8*p6, 3.11270408163265e-7*p5 + 1.0*w2]])


# AutoGenU for Jupyter

An Automatic C++ Code Generator for Nonlinear Model Predictive Control (NMPC)

---

## Formulation of the optimal control problem: Hexacopter

### Model description
Let $(X, Y, Z)$ be a position of the center of the hexiacopter in the world frame, $(\phi, \theta, \psi)$ be the Euler angle of the hexacopter in the world frame, respectively. We assume that we can control the thrust of the each rotor directly. Thus, we assume that the $i$th control input $u_i$ is the thrust of the corresponding rotor. The equation of the motion is derived by the Newton-Euler equations as
$$ \begin{bmatrix} \ddot{X} \\ \ddot{Y} \\ \ddot{Z} \\ \ddot{\phi} \\ \ddot{\theta} \\ \ddot{\psi} \end{bmatrix} = \begin{bmatrix} (\cos{\psi} \sin{\theta} \cos{\phi} + \sin{\psi} \sin{\phi}) \frac{U_1}{m} \\ (\sin{\psi} \sin{\theta} \cos{\phi} - \cos{\psi} \sin{\phi}) \frac{U_1}{m} \\ -g + (\cos{\theta} \cos{\phi}) \frac{U_1}{m}  \\ \frac{I_{YY} - I_{ZZ}}{I_{XX}} \dot{\theta} \dot{\psi} + \frac{U_2}{I_{XX}} \\ \frac{I_{ZZ} - I_{XX}}{I_{YY}} \dot{\psi} \dot{\phi} + \frac{U_3}{I_{YY}} \\ \frac{I_{XX} - I_{YY}}{I_{ZZ}} \dot{\phi} \dot{\theta} + \frac{U_4}{I_{ZZ}} \end{bmatrix} , $$
where
$$ U_1 = \sum_{i=1}^{6} u_i ,$$
$$ U_2 = l(-\frac{u_1}{2} - u_2 - \frac{u_3}{2} + \frac{u_4}{2} + u_5 + \frac{u_6}{2} ) ,$$
$$ U_3 = l(- \frac{\sqrt{3}}{2} u_1 + \frac{\sqrt{3}}{2} u_3 + \frac{\sqrt{3}}{2} u_4 - \frac{\sqrt{3}}{2} u_6) , $$
$$ U_4 = k(-u_1 + u_1 - u_2 + u_3 - u_4 + u_5) - \gamma \dot{\psi} .$$
We also assume that the thrust of the each rotor is limited as
$$u_{\rm min} \leq u_i \leq u_{\rm max} , i= 1, ..., 6,  $$
where $u_{\rm min}, u_{\rm max} \in \mathbb{R}$.

### Formulation of the optimal control problem
We define the state vector as
$$ x = \begin{bmatrix} {X} & {Y} & {Z} & {\phi} & {\theta} & {\psi} & \dot{X} & \dot{Y} & \dot{Z} & \dot{\phi} & \dot{\theta} & \dot{\psi} \end{bmatrix}^{\rm T} .$$
The state equation is then given as
$$\dot{x} = f(t, x, u) = \begin{bmatrix} \dot{X} \\ \dot{Y} \\ \dot{Z} \\ \dot{\phi} \\ \dot{\theta} \\ \dot{\psi} \\ (\cos{\psi} \sin{\theta} \cos{\phi} + \sin{\psi} \sin{\phi}) \frac{U_1}{m} \\ (\sin{\psi} \sin{\theta} \cos{\phi} - \cos{\psi} \sin{\phi}) \frac{U_1}{m} \\ -g + (\cos{\theta} \cos{\phi}) \frac{U_1}{m}  \\ \frac{I_{YY} - I_{ZZ}}{I_{XX}} \dot{\theta} \dot{\psi} + \frac{U_2}{I_{XX}} \\ \frac{I_{ZZ} - I_{XX}}{I_{YY}} \dot{\psi} \dot{\phi} + \frac{U_3}{I_{YY}} \\ \frac{I_{XX} - I_{YY}}{I_{ZZ}} \dot{\phi} \dot{\theta} + \frac{U_4}{I_{ZZ}} \end{bmatrix} .$$
In this problem, we consider the inequality constraints using the barrier function method. In the barrier function method, an inequality constraint
$$ C(t, x, u) \geq 0 $$
is considered by adding the following berrier function to the cost function
$$ - \epsilon \ln\left(\frac{1}{C(t, x, u)}\right), $$
where $\epsilon$ is a sufficient small positive constant. In this problem, we add barrier functions given by
$$ b(u) :=  - \epsilon \sum_{i=1}^{6} \ln\left(\frac{1}{u_{\rm max} - u_i}\right) - \epsilon \sum_{i=1}^{6} \ln\left(\frac{1}{u_i - u_{\rm min}}\right)  $$
to the stage cost in the cost function. Next, we design the cost function
$$ J = \varphi(t, x) + \int_{t}^{t+T} L(t, x, u) .$$
The objective is to make the hexacopter track a reference trajectory. We assume that the reference trajectory is given as
$$ p_{\rm ref} (t) := \begin{bmatrix} X_{\rm ref} (t) \\ Y_{\rm ref} (t) \\ Z_{\rm ref} (t) \end{bmatrix} = \begin{bmatrix} \sin{2t} \\ 1-\cos{2t} \\ z_0 + 2 \sin{t} \end{bmatrix} , $$
where $z_0$ is a positive real constant. We define a time-dependent vector valued function
$$ x_{\rm ref} (t) := \begin{bmatrix} p_{\rm ref} (t) \\ \dot{p}_{\rm ref} (t) \\ 0_3 \\ 0_3 \end{bmatrix} , $$
where $0_3$ is a 3-dimensional zero vector. To make the hexacopter track $p_{\rm ref}$, we set the terminal cost as
$$\varphi(t,x) = \frac{1}{2} (x - x_{\rm ref}(t))^{\rm T} Q_{\rm terminal} (x - x_{\rm ref}(t)) ,$$
where $Q_{\rm terminal} = {\rm diag}\left\{ {q_{\rm terminal}}_1, ... ,{q_{\rm terminal}}_{12} \right\}$ and ${q_{\rm terminal}}_1, ..., {q_{\rm terminal}}_{12}$ are positive real constants. We also set the stage cost as
$$L(t, x, u) = \frac{1}{2} (x - x_{\rm ref}(t))^{\rm T} Q (x - x_{\rm ref}) + \frac{1}{2} (u - u_{\rm ref}(t))^{\rm T} R (x - u_{\rm ref}) + b(u),$$
where $Q = {\rm diag} \left\{ q_1, ..., q_{12} \right\}$ with positive real constants $q_1, ..., q_{12}$, $R = {\rm diag} \left\{ r_1, ..., r_{6} \right\}$ with positive real constants $r_1, ..., r_{6}$, $u_{\rm ref} \in \mathbb{R}^6$ is given by
$$ u_{\rm ref} := \begin{bmatrix} \frac{mg}{6} \\ \frac{mg}{6} \\ \frac{mg}{6} \\ \frac{mg}{6} \\ \frac{mg}{6} \\ \frac{mg}{6} \end{bmatrix} .$$

### Numerical solver of NMPC
In the following example, we compute the solution of NMPC using the multiple shooting based C/GMRES method with condensing variables with respect to the state.

### Reference
The derivative of the model of the hexacopter is reference to [1] and [2]. The paremeters of the hexacopter in the following numerical simulation are based on [1].
1. [Y. Aoki et.al., Nonlinear Model Predictive Control of Position and Attitude in a Hexacopter with Three Failed Rotors, IFAC-PapersOnLine, Vol. 51, No. 20, pp. 228-233 (2018)](https://doi.org/10.1016/j.ifacol.2018.11.018)
2. [M. Moussid et.al., Dynamic Modeling and Control of a HexaRotor using Linear and Nonlinear Methods, International Journal of Applied Information Systems, Vol. 9, No. 5, pp. 9-17 (2015)](https://doi.org/10.5120/ijais2015451411)

---

# Generate C++ codes of NMPC model

## Import modules

```python
from autogenu import autogenu
```

## Set dimensions and define `AutoGenU`
- `dimx`: Dimension of the state vector $x$
- `dimu`: Dimension of the control input vector $u$
- `model_name`: Name of the model. Used as the name of the directory containing the generated C++ source files.

```python
dimx = 12
dimu = 6
model_name = 'hexacopter'
ag = autogenu.AutoGenU(model_name, dimx, dimu)
```

## Generate t, x, and u, necessary variables to formulate of the optimal control problem
- `t`: Time parameter $t$
- `x`: The state vector $x$
- `u`: The control input vector $u$

```python
t = ag.define_t()
x = ag.define_x()
u = ag.define_u()
```

## Define variables used in the state equation, constraints, and cost function
- Define a scalar variable whose name is "var" as
`var = ag.define_scalar_var('var') `

- Define scalar variables whose names are "var\_1", "var\_2", ..., "var\_n" as
`var_1, var_2, ..., var_n = ag.define_scalar_vars('var_1', 'var_2', ..., 'var_n')`

- Define array variables whose name is "vec" and dimension is dim_vec as
`vec = ag.define_array_var('vec', dim_vec)`

```python
# Variables used in the state function and cost function
m, l, k, Ixx, Iyy, Izz, gamma, g = ag.define_scalar_vars('m', 'l', 'k', 'Ixx', 'Iyy', 'Izz', 'gamma', 'g')

# Variables used in the cost function
z_ref, u_min, u_max, epsilon = ag.define_scalar_vars('z_ref', 'u_min', 'u_max', 'epsilon')
q = ag.define_array_var('q', dimx)
q_terminal = ag.define_array_var('q_terminal', dimx)
r = ag.define_array_var('r', dimu)
```

## Define the state equation, constraints, the stage cost, and the terminal cost
- `f`: The state equation $ f(t, x, u)$
- `C`: The equality constraisnts $C(t, x, u) = 0 $
- `h`: The inequality constraisnts $h(t, x, u) \leq 0$ considered by semi-smooth Fischer-Burumeister method
- `L`: The stage cost $L(t, x, u)$
- `phi`: The terminal cost $\phi (t, x)$

Note: array indices start with 0

```python
# Import symbolic math from sympy. If you need more functions, import same as below.
from sympy import sin, cos, tan, exp, log, ln, sinh, cosh, tanh, diff, sqrt
```

```python
# Time-variant reference trajectory
xyz_ref = [sin(2*t), (1-cos(2*t)), z_ref + 2*sin(t)]
xyz_ref_diff = [diff(xyz_ref[i], t) for i in range(len(xyz_ref))]

# propeller's input transformation
U1 = sum(u[i] for i in range(dimu))
U2 = l*(-u[0]/2 - u[1] - u[2]/2 + u[3]/2 + u[4]+ u[5]/2)
U3 = l*(-(sqrt(3)/2)*u[0] + (sqrt(3)/2)*u[2] + (sqrt(3)/2)*u[3] - (sqrt(3)/2)*u[5])
U4 = k*(-u[0] + u[1] - u[2] + u[3] - u[4] + u[5]) - gamma * x[11]

# Define the state equation
f = [x[6],
     x[7],
     x[8],
     x[9],
     x[10],
     x[11],
     (cos(x[5])*sin(x[4])*cos(x[3]) + sin(x[5])*sin(x[3]))*U1/m,
     (sin(x[5])*sin(x[4])*cos(x[3]) - cos(x[5])*sin(x[3]))*U1/m,
      -g + (cos(x[3])*cos(x[4]))*U1/m,
     ((Iyy-Izz)/Ixx)*x[10]*x[11] + U2/Ixx,
     ((Izz-Ixx)/Iyy)*x[9]*x[11] + U3/Iyy,
     ((Ixx-Iyy)/Izz)*x[9]*x[10] + U4/Izz]

# Define the constraints (if dimc > 0)
C = []

# Define the inequality constraints h(x, u) <= 0 (if dimh > 0) considered using the semi-smooth Fischer-Burmeister method
h = []

# Define the stage cost
u_ref = (m*g)/6
u_barrier = sum(-ln(u[i]-u_min) - ln(u_max-u[i]) for i in range(dimu))
L = sum((q[i]*(x[i]-xyz_ref[i])**2)/2 for i in range(3)) + sum((q[i]*x[i]**2)/2 for i in range(3, 6))+ sum((q[i+6]*(x[i+6]-xyz_ref_diff[i])**2)/2 for i in range(3)) + sum((q[i]*x[i]**2)/2 for i in range(9, 12)) + sum(r[i] * (u[i]-u_ref)**2 for i in range(dimu))/2 + epsilon * u_barrier

# Define the terminal cost
phi = sum((q_terminal[i]*(x[i]-xyz_ref[i])**2)/2 for i in range(3)) + sum((q_terminal[i]*x[i]**2)/2 for i in range(3, 6))+ sum((q_terminal[i+6]*(x[i+6]-xyz_ref_diff[i])**2)/2 for i in range(3)) + sum((q_terminal[i]*x[i]**2)/2 for i in range(9, 12))
```

```python
# Set functions
ag.set_functions(f, C, h, L, phi)
```

## Set Parameters
Set values for all parameters you defined.
- Set `value` for a scalar variable whose name is "var" as
`ag.set_scalar_var('var', value) `

- Set `value_1`, `value_2`, ..., `value_n`, for scalar variables whose names are "var\_1", "var\_2", ..., "var\_n" as
`ag.set_scalar_vars(['var_1', value_1], ['var_2', value_2], ..., ['var_n', value_n])`

- Set array variables `var_1`, `var_2`, ..., `var_n` whose name is "vec" and dimension is n as
`ag.define_array_var('vec', [var_1, var_2, ..., var_n])`

```python
ag.set_scalar_vars(['m', 1.44], ['l', 0.23], ['k', 1.6e-09],
                   ['Ixx', 0.0348], ['Iyy', 0.0459], ['Izz', 0.0977],
                   ['gamma', 0.01], ['g', 9.80665],
                   ['z_ref', 5], ['u_min', 0.144], ['u_max', 6], ['epsilon', 0.01])
ag.set_array_var('q', [1, 1, 1, 0.01, 0.01, 0, 0.01, 0.01, 0.01, 0.1, 0.1, 0.001])
ag.set_array_var('r', [0.01, 0.01, 0.01, 0.01, 0.01, 0.01])
ag.set_array_var('q_terminal', [1, 1, 1, 0.01, 0.01, 0, 0.01, 0.01, 0.01, 0.1, 0.1, 0.001])
```

## Generate C++ codes of NMPC model
Generate `nmpc_model.hpp` and `nmpc_model.cpp`, C++ source files of NMPC problem settings.
- `use_simplification`: The flag for simplification. If `True`, symbolic functions are simplified. However, if functions are too complicated, it takes too much time. Default is `False`.
- `use_case`: The flag for common subexpression elimination. If `True`, common subexpressions in fxu, phix, hx, and hu are eliminated when `nmpc_model.cpp` is generated. Default is `False`.

```python
use_simplification = False
use_cse = True

ag.generate_source_files(use_simplification, use_cse)
```

---
# Generate C++ codes for numerical simulation

## Set solvers
Set which solvers you use.
- `ContinuationGMRES` : The continuation/GMRES method (the original C/GMRES method, single shooting).
- `MultipleShootingCGMRES` : The multiple shooting based C/GMRES method with condensing of the state and the Lagragne multipliers with respect to the state equation.
- `MSCGMRESWithInputSaturation` : The multiple shooting based C/GMRES method with condensing of the state, the Lagragne multipliers with respect to the state equation, and variables with respect to the constraints on the saturation function on the control input.

Note that if you have to define the solver type as `autogenu.SolverType.ContinuationGMRES`, `autogenu.SolverType.MultipleShootingCGMRES`, or `autogenu.SolverType.MSCGMRESWithInputSaturation`

```python
solver_type = autogenu.SolverType.MultipleShootingCGMRES
ag.set_solver_type(solver_type)
```

## Set saturaions on the control input if you choose `MSCGMRESWithInputSaturation`
- Saturation on the control input: $u_{i, {\rm min}} \leq u_i \leq u_{i, {\rm max}}$
$u_i \in \mathbb{R}$ : a constrained component of the control input $u$
- Transformed equality constraint: $(u_i - \frac{u_{i, {\rm max}} + u_{i, {\rm min}}}{2})^2 - ( \frac{u_{i, {\rm max}} - u_{i, {\rm min}}}{2})^2 + {u_d}_i ^2 = 0$
${u_d}_i \in \mathbb{R}$ : a dummy input for the transformation of the saturation into the equality constraint
- Additional term in the stage cost $L(x, u)$ with respect to the saturation of $u_i$: $- {r_d}_i {u_d}_i + \frac{1}{2} {r_q}_i {u_d}_i ^2$
   ${r_d}_i > 0$: a weight parameter to avoid failure of numerical computation, ${r_q}_i \geq 0$: a weight parameter to increase mergin of the saturation

- Parameters:
    - `index`: $i$
    - `u_min`: $u_{i, {\rm min}}$
    - `u_max`: $u_{i, {\rm max}}$
    - `dummy_weight`: ${r_d}_i > 0$
    - `quadratic_weight` :  ${r_q}_i \geq 0$

```python
# Add saturation_list as follows.
# ag.add_control_input_saturation(index , u_min, u_max, dummy_weight, quadratic_weight)
```

## Set parameters for the C/GMRES method

- `T_f`, `alpha`: Parameters for the length of the horizon $T(t)$: $T(t) = T_f (1 - e^{-\alpha t})$. The length of the horizon converges to `T_f` and the speed of the convergence is determied by `alpha`.
- `N`: The division number of the horzion for the numerical computation.
- `finite_difference_increment`: Step length of a finite difference approximations of hessian-vector products in C/GMRES.
- `zeta`: A stabilization parameter of the C/GMRES method. Basically, must be set by the reciprocal of the sampling time.
- `kmax`: The maximam number of the iteration of the GMRES.

```python
T_f = 1.0
alpha = 1.0
N = 50
finite_difference_increment = 1.0e-08
zeta = 1000  # zeta must be the reciprocal of the sampling period
kmax = 10

ag.set_solver_parameters(T_f, alpha, N, finite_difference_increment, zeta, kmax)
```

## Set parameters for the initialization of the solution
- `solution_initial_guess`: The initial guess of the solution of the optimal control problem (OCP) for initialization of the solution of NMPC.
- `newton_residual_torelance`: The residual torelance of the solution of the OCP for the initialization of the solution of NMPC. The Newton iteration terminates when the optimality error is less than this velue.
- `max_newton_iteration`: The maxmum number of Newton iteration for the initialization of the solution of NMPC.
- `initial_Lagrange_multiplier`: An optional parameter for `MSCGMRESWithInputSaturation`. This is a part of the initial guess of the solution, the initial guess of the Lagrange multiplier with respect the constraints on the saturation function of the control input. NOTE that this parameter has to be defined as list. If you do not specify this parameter, the all of initial Lagrange multiplier are set by 1e-03.

```python
solution_initial_guess = [1, 1, 1, 1, 1, 1]
newton_residual_torelance = 1.0e-06
max_newton_iteration = 50

ag.set_initialization_parameters(solution_initial_guess, newton_residual_torelance, max_newton_iteration)
```

## Set parameters for numerical simulation
- `initial_time`: Initial time of the numerical simulation.
- `initial_state`: Initial state vector of the system.
- `simulation_time`: Simulation time of the numerical simulation.
- `sampling_time`: The sampling time of the numerical simulation.

```python
initial_time = 0
initial_state = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
simulation_time = 10
sampling_time = 0.001

ag.set_simulation_parameters(initial_time, initial_state, simulation_time, sampling_time)
```

## Generate main.cpp and CMakeLists.txt

```python
ag.generate_main()
ag.generate_cmake()
```

## Build and run simulation

- `generator`: An optional variable for Windows user to choose the generator of CMake. If 'MSYS', then 'MSYS Makefiles' is used. If 'MinGW', then 'MinGW Makefiles' is used. The default value is 'Auto' and the generator is selected automatically. If sh.exe exists in your PATH, MSYS is choosed, and otherwise MinGW is used.
- `remove_build_dir`: If `True`, the existing build directory is removed. If `False`, the build directory is not removed. Need to be set `True` if you change the generator. Default is `False`.

```python
generator = 'Auto'
remove_build_dir = False
ag.build(generator, remove_build_dir)
ag.run_simulation()
```

## Plot the simulation results

```python
import autogenu.simulation_plottor as simplot
```

```python
plottor = simplot.SimulationPlottor(model_name)
plottor.set_scales(2,5,3)
#plottor.show_plots()
plottor.save_plots()
```

## Draw animations of the simulation results

```python
import autogenu.animation_generator as animgen
```

```python
anim = animgen.Hexacopter(model_name)
anim.set_skip_frames(10)
anim.generate_animation()
```