# ML-Fundamentals - Lineare Regression - Exercise: Multivariate Linear Regression

## Table of Contents
* [Introduction](#Introduction)
* [Requirements](#Requirements)
  * [Knowledge](#Knowledge)
  * [Modules](#Python-Modules)
* [Exercises - Multivariate Linear Regression](#Exercises---Multivariate-Linear-Regression)
  * [Create Features](#Create-Features)
  * [Linear Hypothesis](#Linear-Hypothesis)
  * [Generate Target Values](#Generate-Target-Values)
  * [Plot The Data](#Plot-The-Data)
  * [Cost Function](#Cost-Function)
  * [Gradient Descent](#Gradient-Descent)
  * [Training and Evaluation](#Training-and-Evaluation)
  * [Feature Scaling](#Feature-Scaling)
* [Summary and Outlook](#Summary-and-Outlook)
* [Literature](#Literature)
* [Licenses](#Licenses)

## Introduction

In this exercise you will implement the _multivariate linear regression_, a model with two or more predictors and one response variable (opposed to one predictor using univariate linear regression). The whole exercise consists of the following steps:

1. Generate values for two predictors/features $(x_1, x_2)$
2. Implement a linear function as hypothesis (model)
3. Generate values for the response (Y / target values)
4. Plot the $((x_1, x_2), y)$ values in a 3D plot.
5. Write a function to quantify your model (cost function)
6. Implement the gradient descent algorithm to train your model (optimizer)
7. Visualize your training process and results
8. Apply feature scaling (pen & paper)

## Requirements
### Knowledge

You should have a basic knowledge of:
- Univariate linear regression
- Multivariate linear regression
- Squared error
- Gradient descent
- numpy
- matplotlib

Suitable sources for acquiring this knowledge are:
- [Multivariate Linear Regression Notebook](http://christianherta.de/lehre/dataScience/machineLearning/basics/multivariate_linear_regression.php) by Christian Herta and his [lecture slides](http://christianherta.de/lehre/dataScience/machineLearning/multivariateLinearRegression.pdf) (German)
- Chapter 2 of the open classroom [Machine Learning](http://openclassroom.stanford.edu/MainFolder/CoursePage.php?course=MachineLearning) by Andrew Ng
- Chapter 5.1 of [Deep Learning](http://www.deeplearningbook.org/contents/ml.html) by Ian Goodfellow
- Some parts of chapter 1 and 3 of [Pattern Recognition and Machine Learning](https://www.microsoft.com/en-us/research/people/cmbishop/#!prml-book) by Christopher M. Bishop
- [numpy quickstart](https://docs.scipy.org/doc/numpy-1.15.1/user/quickstart.html)
- [Matplotlib tutorials](https://matplotlib.org/tutorials/index.html)

### Python Modules

By [deep.TEACHING](https://www.deep-teaching.org/) convention, all python modules needed to run the notebook are loaded centrally at the beginning.

```python
# External Modules
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d.axes3d import Axes3D
from matplotlib import cm

%matplotlib notebook
```

## Exercise - Multivariate Linear Regression

We will only use two features in this notebook, so we are still able to plot them together with the target in a 3D plot. But your implementation should also be capable of handling more (except the plots).

### Create Features

First we will create some features. The features should be in a 2D numpy array, the rows separating the different feature vectors (training examples), the columns containing the features. Each feature should be **uniformly** distributed in a specifiable range.

**Task:**

Implement the function to generate a feature matrix (numpy array).

```python
def create_feature_matrix(sample_size, n_features, x_min, x_max):
    '''creates random feature vectors based on a lienar function in a given interval

    Args:
        sample_size: number feature vectors
        n_features: number of features for each vector
        x_min: lower bound value ranges
        x_max: upper bound value ranges

    Returns:
        x: 2D array containing feature vecotrs with shape (sample_size, n_features)
    '''
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

def create_feature_matrix(sample_size, n_features, x_min, x_max):
    '''creates random feature vectors based on a lienar function in a given interval

    Args:
        sample_size: number feature vectors
        n_features: number of features for each vector
        x_min: lower bound value ranges
        x_max: upper bound value ranges

    Returns:
        x: 2D array containing feature vecotrs with shape (sample_size, n_features)
    '''

    return np.random.uniform(x_min, x_max, (sample_size, n_features))
```

```python
sample_size = 100
n_features = 2
x_min = [1.5, -0.5]
x_max = [11., 5.0]

X = create_feature_matrix(sample_size, n_features, x_min, x_max)
X
```

    array([[ 2.55343578,  4.09882663],
           [ 9.14676162,  1.75409426],
           [ 6.87061128,  3.088036  ],
           [ 3.31754536, -0.10581374],
           [10.04676143,  4.91327452],
           [ 9.53248102, -0.21669548],
           [ 4.74762432,  2.11592889],
           [ 3.04610114,  0.66861665],
           [ 8.99810878,  4.10601558],
           [ 2.03046397,  1.35255697],
           [ 4.6004498 ,  1.01205624],
           [ 5.2053511 ,  4.85018639],
           [ 8.92142801,  3.40039974],
           [ 8.41672928,  3.66919872],
           [ 5.01825985,  2.51420676],
           [ 9.08429314,  1.04931984],
           [ 2.15416736,  0.5268242 ],
           [ 6.2284124 ,  4.54399381],
           [ 8.10072993,  1.84264001],
           [ 3.71944741,  0.94380792],
           [ 4.35883408,  1.26337124],
           [ 4.43690719,  0.99997779],
           [ 2.51908645,  0.34135765],
           [ 4.64194197,  1.37604907],
           [ 5.56066631,  4.24084101],
           [ 3.31660389,  3.4357965 ],
           [ 7.46692545, -0.13872683],
           [ 8.67304278,  1.53969574],
           [ 2.25378335,  2.1360613 ],
           [ 2.6540387 ,  1.03259774],
           [ 5.02662661,  2.87918316],
           [ 4.12726842,  4.23028426],
           [ 5.68731577,  2.70034396],
           [10.74272894, -0.16190299],
           [ 5.57938986,  3.30598652],
           [ 9.26774676,  3.15054548],
           [ 4.23662404,  4.93963246],
           [ 3.84702751,  0.70003255],
           [ 7.23072073,  2.14386805],
           [ 1.57051795,  3.70768626],
           [ 5.40511937,  3.34361318],
           [ 1.86679871,  4.50691358],
           [ 2.64773238,  3.48214973],
           [ 5.29444443,  1.06644367],
           [ 8.15360051,  1.91813851],
           [ 1.58028104,  3.52942058],
           [ 5.57410943,  0.36621873],
           [ 3.51710302,  4.05378562],
           [ 1.9668387 ,  3.08308516],
           [ 5.59368553,  0.48926087],
           [ 4.72382896,  1.27217418],
           [ 4.75903772,  4.02492114],
           [10.32314113,  3.45291417],
           [ 8.3607832 ,  1.41331422],
           [ 5.02277249, -0.17925784],
           [ 4.96284139,  3.16654885],
           [ 5.25042165,  4.12198925],
           [ 3.87229428,  2.14576293],
           [ 9.53374907, -0.05896827],
           [ 8.15028007,  3.9327573 ],
           [ 9.56958557,  2.32766841],
           [ 7.64470758,  1.88364085],
           [ 9.19808575,  2.63258445],
           [ 8.54669469,  2.58779554],
           [ 2.0067687 ,  3.09877182],
           [ 3.62733662,  0.25622579],
           [10.05874243,  0.19438902],
           [ 8.14099486,  4.01139473],
           [ 7.21657038,  2.4667673 ],
           [ 8.1279165 ,  0.24171846],
           [ 1.85488792,  2.11330979],
           [ 6.03449596,  3.593834  ],
           [ 3.28212156, -0.2277349 ],
           [ 7.08842704,  2.76401252],
           [ 8.22070264,  3.87316012],
           [ 7.86751765,  2.55121306],
           [ 3.23420664,  4.64625445],
           [ 2.26114604,  3.59172898],
           [ 9.69223623,  1.5591381 ],
           [ 8.4105376 ,  4.45931284],
           [ 2.22748906,  2.79878555],
           [ 7.32528596,  2.07677038],
           [10.36156224,  3.67408725],
           [ 6.55661923, -0.46412987],
           [ 3.9209253 , -0.47884172],
           [ 8.8067252 ,  4.58892015],
           [ 4.13219722, -0.30789565],
           [10.96105526,  2.83739255],
           [ 6.83442425,  2.91034924],
           [ 9.49990468, -0.17566033],
           [ 6.95625609,  1.48659351],
           [ 4.37699785,  1.08344818],
           [ 2.00884202,  0.23834419],
           [ 3.33201506,  2.12857654],
           [ 1.50306421,  0.71705611],
           [ 9.46405326,  1.78328114],
           [ 9.61235747,  1.24012355],
           [ 1.96521893,  0.76458039],
           [ 6.35091656,  4.27638952],
           [ 6.44543344,  2.78178998]])

```python
assert len(X[:,0]) == sample_size
assert len(X[0,:]) == n_features
for i in range(n_features):
    assert np.max(X[:,i]) <= x_max[i]
    assert np.min(X[:,i]) >= x_min[i]
```

### Linear Hypothesis

A short recap, a hypothesis $h_\theta({\bf x})$ is a certain function that we believe is similar to a target function that we like to model. A hypothesis $h_\theta({\bf x})$ is a function of ${\bf x}$ with fixed parameters $\Theta$.

Here we have $n$ features ${\bf x} = \{x_1, \ldots, x_n \}$ and $n+1$ parameters $\Theta = \{\theta_0, \theta_1 \ldots, \theta_n \}$:

$$
h_\theta({\bf x}) = \theta_{0} + \theta_{1} x_1 + \ldots \theta_n x_n
$$

adding an extra element to $\vec x$ for convenience, this could also be rewritten as:

$$
h_\theta({\bf x}) = \theta_{0} x_0 + \theta_{1} x_1 + \ldots \theta_n x_n
$$

with $x_0 = 1$ for all feature vectors (training examples).

Or treating ${\bf x}$ and $\Theta$ as vectors:

$$
h(\vec x) = \vec x'^T \vec \theta
$$

with:

$$
\vec x = \begin{pmatrix}
x_1 & x_2 & \ldots & x_n \\
\end{pmatrix}^T
\text{   and   }
\vec x' = \begin{pmatrix}
1 & x_1 & x_2 & \ldots & x_n \\
\end{pmatrix}^T
$$

and

$$
\vec \theta = \begin{pmatrix}
\theta_0 & \theta_1 & \ldots & \theta_n \\
\end{pmatrix}^T
$$

Or for the whole data set at once: The rows in $X$ separate the different feature vectors, the columns contain the features.

$$
\vec h_\Theta(X) = X' \cdot \vec \theta
$$

the vector $\vec h(X) = \left( h(\vec x^{(1)}),h(\vec x^{(2)}), \dots, h(\vec x^{(m)}) )\right)^T$ contains all predictions for the data batch $X$.

with:

$$
\begin{align}
X &= \begin{pmatrix}
x_1^{(1)} & \ldots & x_n^{(1)} \\
x_1^{(2)} & \ldots & x_n^{(2)} \\
\vdots &\vdots &\vdots \\
x_1^{(m)} & \ldots & x_n^{(m)} \\
\end{pmatrix}
&=
\begin{pmatrix}
\vec x^{(1)T} \\
\vec x^{(2)T}  \\
\vdots  \\
\vec x^{(m)T}  \\
\end{pmatrix}
\end{align}
$$
respectively
$$
\begin{align}
X' = \begin{pmatrix}
1 & x_1^{(1)} & \ldots & x_n^{(1)} \\
1 & x_1^{(2)} & \ldots & x_n^{(2)} \\
\vdots &\vdots &\vdots &\vdots \\
1 & x_1^{(m)} & \ldots & x_n^{(m)} \\
\end{pmatrix}
&=
\begin{pmatrix}
\vec x'^{(1)T} \\
\vec x'^{(2)T}  \\
\vdots  \\
\vec x'^{(m)T}  \\
\end{pmatrix}
\end{align}
$$

**Task:**

Implement hypothesis $\vec h_\Theta(X)$ in the method `linear_hypothesis` and return it as a function. Implement it the computationally efficient (**pythonic**) way by not using any loops and handling all data at once (use $X$ respectively $X'$).

**Hint:**

Of course you are free to implement as many helper functions as you like, e.g. for transforming $X$ to $X'$, though you do not have to. Up to you.

```python
def linear_hypothesis(thetas):
    ''' Combines given list argument in a linear equation and returns it as a function

    Args:
        thetas: list of coefficients

    Returns:
        lambda that models a linear function based on thetas and x
    '''
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

def linear_hypothesis(thetas):
    ''' Combines given list argument in a linear equation and returns it as a function

    Args:
        thetas: list of coefficients

    Returns:
        lambda that models a linear function based on thetas and x
    '''
    return lambda x: np.concatenate((np.ones((len(x[:,0]), 1)), x), axis=1).dot(thetas)
```

```python
assert len(linear_hypothesis([.1,.2,.3])(X)) == sample_size
```

### Generate Target Values

**Task:**

Use your implemented `linear_hypothesis` inside the next function to generate some target values $Y$. Additionally add some Gaussian noise.

```python
def generate_targets(X, theta, sigma):
    ''' Combines given arguments in a linear equation with X,
    adds some Gaussian noise and returns the result

    Args:
        X: 2D numpy feature matrix
        theta: list of coefficients
        sigma: standard deviation of the gaussian noise

    Returns:
        target values for X
    '''
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

def generate_targets(X, theta, sigma):
    ''' Combines given arguments in a linear equation with X,
    adds some Gaussian noise and returns the result

    Args:
        X: 2D numpy feature matrix
        theta: list of coefficients
        sigma: standard deviation of the gaussian noise

    Returns:
        target values for X
    '''
    noise = np.random.randn(len(X)) * sigma
    h = linear_hypothesis(theta)
    y = h(X) + noise
    return y
```

```python
theta = (2., 3., -4.)
sigma = 3.
y = generate_targets(X, theta, sigma)
```

```python
assert len(y) == sample_size
```

### Plot The Data

**Task:**

Plot the data $\mathcal D = \{((x^{(1)}_1,x^{(1)}_2)^T,y^{(1)}), \ldots, ((x^{(n)}_1,x^{(n)}_2)^T,y^{(n)})\}$ in a 3D scatter plot. The plot should look like the following:

**Sidenote:**

The command `%matplotlib notebook` (instead of `%matplotlib inline`) creates an interactive (e.g. rotatable) plot.

```python
%matplotlib notebook

def plot_data_scatter(features, targets):
    """ Plots the features and the targets in a 3D scatter plot

    Args:
        features: 2D numpy-array features
        targets: ltargets
    """
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

%matplotlib notebook

def plot_data_scatter(features, targets):
    """ Plots the features and the targets in a 3D scatter plot

    Args:
        features: 2D numpy-array features
        targets: ltargets
    """
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.scatter(features[:,0], features[:,1], targets, c='r')
    ax.set_xlabel('$x_1$')
    ax.set_ylabel('$x_2$')
    ax.set_zlabel('$y$')
```

```python
plot_data_scatter(X, y)
```

    <IPython.core.display.Javascript object>

### Cost Function
A cost function $J$ depends on the given training data $D$ and hypothesis $h_\theta(\vec x)$. In the context of the linear regression, the cost function measures how "wrong" a model is regarding its ability to estimate the relationship between $\vec x$ and $y$ for specific $\Theta$ values. Later we will treat this as an optimization problem and try to minimize the cost function $J_{\mathcal D}(\Theta)$ to find optimal $\theta$ values for our hypothesis $h_\theta(\vec x)$. The cost function we use in this exercise is the [Mean-Squared-Error](https://en.wikipedia.org/wiki/Mean_squared_error) cost function:

\begin{equation}
    J_{\mathcal D}(\Theta)=\frac{1}{2m}\sum_{i=1}^{m}{(h_\Theta(\vec x^{(i)})-y^{(i)})^2}
\end{equation}

Implement the cost function $J_D(\Theta)$ in the method `mse_cost_function`. The method should return a function that takes the values of $\Theta$ as an argument.

As a sidenote, the terms "loss function" or "error function" are often used interchangeably in the field of Machine Learning.

```python
def mse_cost_function(x, y):
    ''' Implements MSE cost function as a function J(theta) on given traning data

    Args:
        x: vector of x values
        y: vector of ground truth values y

    Returns:
        lambda J(theta) that models the cost function
    '''
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

def mse_cost_function(x, y):
    ''' Implements MSE cost function as a function J(theta) on given traning data

    Args:
        x: vector of x values
        y: vector of ground truth values y

    Returns:
        lambda J(theta) that models the cost function
    '''
    assert(len(x) == len(y))
    m = len(x)
    return lambda theta: 1./(2. * float(m)) * np.sum((linear_hypothesis(theta)(x) - y )**2)
```

Review the cell in which you generate the target values and note the theta values, which were used for it (If you haven't edited the default values, it should be `[2, 3, -4]`)

**Optional:**

Try a few different values for theta to pass to the cost function - Which thetas result in a low error and which produce a great error?

```python
J = mse_cost_function(X, y)
print(J(theta))
```

    4.675479225972847

###  Gradient Descent

A short recap, the gradient descent algorithm is a first-order iterative optimization for finding a minimum of a function. From the current position in a (cost) function, the algorithm steps proportional to the negative of the gradient and repeats this until it reaches a local or global minimum and determines. Stepping proportional means that it does not go entirely in the direction of the negative gradient, but scaled by a fixed value $\alpha$ also called the learning rate. Implementing the following formalized update rule is the core of the optimization process:

\begin{equation}
    \theta_{j}^{new} \leftarrow \theta_{j}^{old} - \alpha * \frac{\partial}{\partial\theta_{j}} J(\vec \theta^{old})
\end{equation}

**Task:**

Implement the function to update all theta values.

```python
def update_theta(x, y, theta, learning_rate):
    ''' Updates learnable parameters theta

    The update is done by calculating the partial derivities of
    the cost function including the linear hypothesis. The
    gradients scaled by a scalar are subtracted from the given
    theta values.

    Args:
        x: 2D numpy array of x values
        y: array of y values corresponding to x
        theta: current theta values
        learning_rate: value to scale the negative gradient

    Returns:
        theta: Updated theta vector
    '''
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

def update_theta(x, y, theta, learning_rate):
    ''' Updates learnable parameters theta

    The update is done by calculating the partial derivities of
    the cost function including the linear hypothesis. The
    gradients scaled by a scalar are subtracted from the given
    theta values.

    Args:
        x: 2D numpy array of x values
        y: array of y values corresponding to x
        theta: current theta values
        learning_rate: value to scale the negative gradient

    Returns:
        theta: Updated theta vector
    '''
    m = len(x[:,0])
    # update rules
    x_ = np.concatenate((np.ones([m,1]), x), axis=1)
    theta = theta  - alpha / float(m) * (x_.T.dot(linear_hypothesis(theta)(x) - y) )

    return theta
```

Using the `update_theta` method, you can now implement the gradient descent algorithm. Iterate over the update rule to find the values for $\vec \theta$ that minimize our cost function $J_D(\vec \theta)$. This process is often called training of a machine learning model.

**Task:**
- Implement the function for the gradient descent.
- Create a history of all theta and cost values and return them.

```python
def gradient_descent(learning_rate, theta, iterations, x, y):
    ''' Minimize theta values of a linear model based on MSE cost function

    Args:
        learning_rate: scalar, scales the negative gradient
        theta: initial theta values
        x: vector, x values from the data set
        y: vector, y values from the data set
        iterations: scalar, number of theta updates

    Returns:
        history_cost: cost after each iteration
        history_theta: Updated theta values after each iteration
    '''
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

def gradient_descent(learning_rate, theta, iterations, x, y):
    ''' Minimize theta values of a linear model based on MSE cost function

    Args:
        learning_rate: scalar, scales the negative gradient
        theta: initial theta values
        x: vector, x values from the data set
        y: vector, y values from the data set
        iterations: scalar, number of theta updates

    Returns:
        history_cost: cost after each iteration
        history_theta: Updated theta values after each iteration
    '''
    history_cost = np.zeros(nb_iterations)
    history_theta = np.zeros([iterations, len(theta)])
    cost = mse_cost_function(x, y);
    for i in range(nb_iterations):
        history_theta[i] = theta
        history_cost[i] = cost(theta)
        theta = update_theta(x, y, theta, learning_rate)
    return history_cost, history_theta
```

### Training and Evaluation

**Task:**

Choose an appropriate learning rate, number of iterations and initial theta values and start the training

```python
# Your implementation:

alpha = 42.42 # assign an appropriate value
nb_iterations = 1337 # assign an appropriate value
start_values_theta = [42., 42., 42.] # assign appropriate values
history_cost, history_theta = gradient_descent(alpha, start_values_theta, nb_iterations, X, y)
```

    <ipython-input-17-bfcd484b6e8e>:15: RuntimeWarning: overflow encountered in square
      return lambda theta: 1./(2. * float(m)) * np.sum((linear_hypothesis(theta)(x) - y )**2)

```python
# Solution

alpha = 0.01
nb_iterations = 100
start_values_theta = [1.,-1.,.5]
history_cost, history_theta = gradient_descent(alpha, start_values_theta, nb_iterations, X, y)
```

Now that the training has finished we can visualize our results.

**Task:**

Plot the costs over the iterations. If you have used `fig = plt.figure()` and `ax = fig.add_subplot(111)` in the last plot, use it again here, else the plot will be added to the last plot instead of a new one.

Your plot should look similar to this one:

```python
def plot_progress(costs):
    """ Plots the costs over the iterations

    Args:
        costs: history of costs
    """
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

def plot_progress(costs):
    """ Plots the costs over the iterations

    Args:
        costs: history of costs
    """
    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.plot(np.array(range(len(costs))), costs)
    ax.set_xlabel('Iterationen')
    ax.set_ylabel('Kosten')
    ax.set_title('Fortschritt')
```

```python
plot_progress(history_cost)
print("costs before the training:\t ", history_cost[0])
print("costs after the training:\t ", history_cost[-1])
```

    <IPython.core.display.Javascript object>

    costs before the training:	  190.45165080692664
    costs after the training:	  4.947880491325002

**Task:**

Finally plot the decision hyperplane (just a plain plane here though) together with the data in a 3D plot.

Your plot should look similar to this one:

```python
def evaluation_plt(x, y, final_theta):
    ''' Plots the data x, y together with the final model

    Args:
        cost_hist: vector, history of all cost values from a opitmization
        theta_0: scalar, model parameter for boundary
        theta_1: scalar, model parameter for boundary
        x: vector, x values from the data set
        y: vector, y values from the data set
    '''
    raise NotImplementedError("You should implement this!")
```

```python
# Solution

def evaluation_plt(x, y, final_theta):
    g = 100
    x_1 = np.linspace(np.min(X[:,0]-1), np.max(X[:,0]+1), g)
    x_2 = np.linspace(np.min(X[:,1]-1), np.max(X[:,1]+1), g)
    X1, X2 = np.meshgrid(x_1, x_2)

    Y = final_theta[0] + final_theta[1] * X1 + final_theta[2] * X2
    print(Y.shape)

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')

    ax.plot_surface(X1, X2, Y, cmap=cm.jet, rstride=5, cstride=5, antialiased=True, shade=True, alpha=0.5, linewidth=0. )
    ax.scatter(x[:,0], x[:,1], y, c='r')
    ax.set_xlabel('$x_1$')
    ax.set_ylabel('$x_2$')
    ax.set_zlabel('$y$')
```

```python

evaluation_plt(X, y, history_theta[-1])
print("thetas before the training:\t", history_theta[0])
print("thetas after the training:\t", history_theta[-1])
```

    (100, 100)

    <IPython.core.display.Javascript object>

    thetas before the training:	 [ 1.  -1.   0.5]
    thetas after the training:	 [ 1.24635143  3.02581154 -3.83837415]

### Feature Scaling

Now suppose the following features $X$:

```python
X = np.array([[0.0001, 2000],
       [0.0002, 1800],
       [0.0003, 1600]], dtype=np.float32)

sample_size = len(X[:,0])
print(X)
```

    [[1.0e-04 2.0e+03]
     [2.0e-04 1.8e+03]
     [3.0e-04 1.6e+03]]

**Task:**

This task can be done via **pen & paper** or by inserting some code below. Either way, you should be able to solve both tasks below on paper only using a calculator.

1. Apply feature scaling onto `X` using the *mean* and the *standard deviation*. What values do the scaled features have?
    * *Optional:*

        You can even execute the cell above and start running your notebook again from top (all **except** executing the cell to generate your features, which would overwrite these new features).

        When you start training you should notice that your costs do not decrease, maybe even increase, if you have not adjusted your learning rate (training might also throw an overflow warning).

**Task:**

2. After the training with scaled features your new $\\vec \theta'$ values will something like: $\vec \theta'=\left(-7197,  326, -326\right)^T$ (you can try training with but you do not have to).

    Suppose $\vec \theta'=\left(-7197,  326, -326\right)^T$. What are the corresponding $\theta_j$ values for the unscaled data?

    * (If you did train your model with the scaled features, the resulting $\theta_j$ should really be $\vec \theta'=\left(-7197,  326, -326\right)^T$

```python
# Solution for 1) + 2)

### Generate new targets for new features X

theta = (2., 3., -4.)
sigma = 3.
y = generate_targets(X, theta, sigma)
```

```python
mu = X.mean(axis=0)
std = X.std(axis=0)
print(mu, std)
```

    [2.0000001e-04 1.8000000e+03] [8.1649661e-05 1.6329932e+02]

```python
X_scaled = (X - mu) / std
X_scaled
```

    array([[-1.2247449e+00,  1.2247449e+00],
           [-1.7822383e-07,  0.0000000e+00],
           [ 1.2247449e+00, -1.2247449e+00]], dtype=float32)

```python
### Calculate mean and standarddeviation and scale X

mu1 = X[:,0].mean()
std1 = np.sqrt(np.var(X[:,0]))
mu2 = X[:,1].mean()
std2 = np.sqrt(np.var(X[:,1]))

print("meanX1:\t", mu1, "\tstdX1:\t", std1)
print("meanX2:\t", mu2, "\tstdX2:\t", std2)

X[:,0] = (X[:,0] - mu1) / std1
X[:,1] = (X[:,1] - mu2) / std2
print("X scaled:\n", X)

```

    meanX1:	 0.00020000001 	stdX1:	 8.164966e-05
    meanX2:	 1800.0 	stdX2:	 163.29932
    X scaled:
     [[-1.2247449e+00  1.2247449e+00]
     [-1.7822383e-07  0.0000000e+00]
     [ 1.2247449e+00 -1.2247449e+00]]

```python
### Train with scaled X

alpha = 0.5
nb_iterations = 100

theta = (2., 3., -4.)
sigma = 3.
y = generate_targets(X, theta, sigma)

start_values_theta = [1.,-1.,.5]
history_cost, history_theta = gradient_descent(alpha, start_values_theta, nb_iterations, X_scaled, y)

print("thetas before the training:\t", history_theta[0])
print("thetas after the training:\t", history_theta[-1])
print("----------------------------")
print("costs before the training:\t", history_cost[0])
print("costs after the training:\t", history_cost[-1])
```

    thetas before the training:	 [ 1.  -1.   0.5]
    thetas after the training:	 [ 1.07576387  2.82165621 -3.32163366]
    ----------------------------
    costs before the training:	 43.92675099402078
    costs after the training:	 14.713938810789191

```python
### Compute theta for unscaled data

theta_hat = np.array([-7197.,  326., -326.])
print("theta_hat:\t", theta_hat)

theta_hat_rescaled = np.array([0.,0.,0.])
theta_hat_rescaled[0] = theta_hat[0] - theta_hat[1]*mu1/std1 - theta_hat[2]*mu2/std2
theta_hat_rescaled[1] = theta_hat[1] / std1
theta_hat_rescaled[2] = theta_hat[2] / std2
print("theta_rescaled:\t", theta_hat_rescaled)

start_values_theta = theta_hat_rescaled
```

    theta_hat:	 [-7197.   326.  -326.]
    theta_rescaled:	 [-4.40213221e+03  3.99266812e+06 -1.99633414e+00]

```python
### Rescale X to original

X = np.array([[0.0001, 2000],
       [0.0002, 1800],
       [0.0003, 1600]], dtype=np.float32)
```

```python
### Test costs for original X and new rescaled theta

alpha = 0.000
nb_iterations = 10
history_cost, history_theta = gradient_descent(alpha, start_values_theta, nb_iterations, X, y)

print("costs with unscaled X and new theta:\t ", history_cost[-1])
```

    costs with unscaled X and new theta:	  26114727.727051158

**Solution 3:**

With polynomila regression features like `1000` would easily skyrocket even more. And worse: `0.00001` would become too small very fast and lose precision. In the worst case values would overflow or become zero.

## Summary and Outlook

During this exercise, the linear regression was extended to multidimensional feature space and feature scaling was practiced. You should be able to answer the following questions:
- How does the implementation of the multivariate regression differ from the univariate one?
- Why do we apply feature scaling?
- Why does feature scaling help?

## Licenses

### Notebook License (CC-BY-SA 4.0)

*The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).*

Exercise: Multivariate Linear Regression <br/>
by Christian Herta, Klaus Strohmenger<br/>
is licensed under a [Creative Commons Attribution-ShareAlike 4.0 International License](http://creativecommons.org/licenses/by-sa/4.0/).<br/>
Based on a work at https://gitlab.com/deep.TEACHING.

### Code License (MIT)

*The following license only applies to code cells of the notebook.*

Copyright 2018 Christian Herta, Klaus Strohmenger

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


# SVM

## What is SVM

[Definition](https://en.wikipedia.org/wiki/Support-vector_machine#Definition): More formally, a support-vector machine constructs a hyperplane or set of hyperplanes in a high- or infinite-dimensional space, which can be used for classification, regression, or other tasks like outliers detection. Intuitively, a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class (so-called functional margin), since in general the larger the margin, the lower the generalization error of the classifier.

With above graph, direct representation of what SVM is tring to do:

$$
\begin{align*}
  &arg \underset{boundary}{max} margin(boundary), \\
  &s.t. \text{the gap between the final boundary and all the correctly classified apples/bananas >= margin} \\
  &\text{(when given a boundary, the margin function calculates the minimal gap between that boundary and all the apples/bananas)}
\end{align*}
$$

## Functional margin vs geometric margin, and SVM interpretation

Lets say we had the final hyperplane: $w^Tx + b = 0$, then we can use $|w^Tx_i + b|$ to represent how far the data point $x_i$ is from the hyperplane, and also to the binary classification problems: we can define that when $(w^Tx_i + b) > 0$ then $x_i \in class 1$, and when $(w^Tx_i + b) < 0$ then $x_i \in class -1$;

With $y \in \{-1, 1\}$, to those correctly classified data points we will always have $y(w^Tx + b) > 0$, and the bigger the value is the more confident we are that the data point is correctly classified;

$\hat{\gamma_i} = y_i(w^Tx_i + b)$ is so called **functional margin** between the data point $(x_i, y_i)$ and the hyperplane $w^Tx + b = 0$;

But there is one problem with this functional margin, when we proportionally scale up/down the $w, b$ to $Mw, Mb$, the hyperplane never got changed, but the original functional margin $\hat{\gamma_i}$ will be changed to $M\hat{\gamma_i}$;

To standardize the margin, there is **geometric margin**:

$$
\gamma_i = y_i \frac{w^Tx_i + b}{||w||}
$$

So another representation of what SVM is trying to achieve:

$$
\begin{align*} \tag{1}
  &arg \, \underset{w, b}{max} \, \gamma \\
  &s.t. \, y_i \frac{w^Tx_i + b}{||w||} \geq \gamma, \forall i
\end{align*}
$$

As the connection between the functional margin $\hat{\gamma_i}$ and the geometric margin $\gamma_i$:

$$\gamma_i = \frac{\hat{\gamma_i}}{||w||}$$

the above representation **(1)** also can be interpreted to:

$$
\begin{align*} \tag{2}
  &arg \, \underset{w, b}{max} \, \frac{\hat{\gamma}}{||w||} \\
  &s.t. \, y_i(w^Tx_i + b) \geq \hat{\gamma}, \forall i
\end{align*}
$$

As we told above that we can always proportionally scale up/down the $w, b$ without affecting the actual hyperplane we are looking for, so for simplifying the problem: we can tweak the $w, b$ to make the functional margin be 1, then further more we can convert the representation to:

$$
\begin{align*} \tag{3}
  &arg \, \underset{w, b}{max} \, \frac{1}{||w||} \\
  &s.t. \, y_i(w^Tx_i + b) \geq 1, \forall i
\end{align*}
$$

In ML we are always accustomed to solve the minimization problem, not the maximization problem, so lets further more convert the above **(3)**:

$$
\begin{align*} \tag{4}
  &arg \, \underset{w, b}{min} \, \frac{1}{2}||w||^2 \\
  &s.t. \, y_i(w^Tx_i + b) \geq 1, \forall i
\end{align*}
$$

This is the final interpretation of what SVM is trying to achieve.

## Lagrange multiplier, Lagrange duality and KKT conditions

### Lagrange equation

With above form **(4)**, lets construct the Lagrange equation:

$$
\begin{equation} \tag{5}
  L(w, b, \alpha)=\frac{1}{2}\|w\|^2 - \sum_{i=1}^n \alpha_{i}\big(y_i(w^T \cdot x_i + b) - 1\big)
\end{equation}
$$

The methodology behind the way constructing the Lagrange equation please refer to this image [lagrange duality](https://github.com/lnshi/ml-exercises/blob/master/ml_basics/rdm011_support_vector_machines/lagrange_duality.png), it is a long image, i don't want to put here, especially pay attention to the red box highlighted part.

###  Primal problem:

$$
\begin{equation} \tag{5.1}
  \min _{w, b} \, \big[ \max _{\alpha : \alpha_{i} \geq 0} L(w, b, \alpha) \big]
\end{equation}
$$

### Dual problem:

$$
\begin{equation} \tag{5.2}
  \max _{\alpha : \alpha_{i} \geq 0} \, \big[ \min _{w, b} L(w, b, \alpha) \big]
\end{equation}
$$

The reason why we need to introduce in the dual problem, and solve the dual problem instead of the primal problem, and a full example of solving a real problem can be found in this image [lagrange duality example](https://github.com/lnshi/ml-exercises/blob/master/ml_basics/rdm011_support_vector_machines/lagrange_duality_example.png), again it is a long image, i don't want to put here, especially pay attention to the red box highlighted part.

In short words: in the primal problem, after we treat $w, b$ as a fixed value, then $L(w, b, \alpha)$ is a **LINEAR FUNCTION** with independent variables $x_i$, we **CANNOT** get its maxima/minima by taking advantages of those derivative/partial derivative knowledges, the only left way is brute force compare, definitely that is NOT what we want. But in the dual problem, we can.

### How to solve the dual problem

$$
\begin{align*}
  &\frac{\partial L}{\partial w}=0 \Longrightarrow w=\sum_{i=1}^n \alpha_i y_i x_i \tag{5.3} \\
  &\frac{\partial L}{\partial b}=0 \Longrightarrow 0=\sum_{i=1}^n \alpha_i y_i \tag{5.4}
\end{align*}
$$

Combine (5.3) with (5) we can get:

$$
\begin{align*} \tag{5.5}
  L(w, b, \alpha) &= \frac{1}{2}\|w\|^{2}-\sum_{i=1}^n \alpha_{i}\big(y_{i}(w^{T} \cdot x_{i}+b)-1\big) \\
  &= \frac{1}{2} w^T w - w^T \sum_{i=1}^n \alpha_i y_i x_i - b\sum_{i=1}^n \alpha_i y_i + \sum_{i=1}^n \alpha_i \\
  &= \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j y_i y_j x_i^T x_j - \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j y_i y_j x_i^T x_j + \sum_{i=1}^n \alpha_i \\
  &= \sum_{i=1}^n \alpha_i - \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j y_i y_j x_i^T x_j
\end{align*}
$$

Finally the dual problem is converted to:

$$
\begin{align*} \tag{5.6}
  &\max _{\alpha} \Bigg[ \sum_{i=1}^n \alpha_i - \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j y_i y_j x_i^T x_j \Bigg] \\
  &s.t. \sum_{i=1}^n \alpha_{i} y_{i} = 0 \\
  &\quad \,\,\, \alpha_i \geq 0, \forall i
\end{align*}
$$

Of course, again you can covert it to a minimization problem:

$$
\begin{align*} \tag{5.7}
  &\min _{\alpha} \Bigg[\frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j y_i y_j x_i^T x_j - \sum_{i=1}^n \alpha_i \Bigg] \\
  &s.t. \sum_{i=1}^n \alpha_{i} y_{i} = 0 \\
  &\quad \,\,\, \alpha_i \geq 0, \forall i
\end{align*}
$$

### KKT conditions

Take the below optimization problem as example:

$$
\begin{align*}
  &min f(x) \\
  &s.t. g_{j}(x) = 0 \, (j=1,2, \cdots, p) \\
  &\quad \,\, h_{k}(x) \leq 0 \, (k=1,2, \cdots, q)
\end{align*}
$$

Construct Lagrange equation:

$$
\begin{align*}
  L(x, \alpha, \beta) = f(x) + \sum_{j=1}^p \alpha_j g_j(x) + \sum_{k=1}^q \beta_k h_k(x)
\end{align*}
$$

KKT conditions are the necessary conditions to make $x^*$ be the optimal solution of the original optimization problem after we did all of the transformations:
1. Lagrange equation construction;
2. Primal problem construction;
3. Dual problem construction;

$$
\begin{cases}
  \frac{\partial f}{\partial x_i} + \sum\limits_{j=1}^p \alpha_j \frac{\partial g_j}{\partial x_i} + \sum\limits_{k=1}^q \beta_k \frac{\partial h_k}{\partial x_i} = 0 &\quad (i = 1, 2, \cdots, m) \\
  g_{j}(x) = 0 &\quad (j=1,2, \cdots, p) \\
  \beta_k h_{k}(x) = 0 &\quad (k=1,2, \cdots, q) \\
  \beta_k \geq 0
\end{cases}
$$

## Coordinate ascent algorithm and SMO

### Coordinate ascent algorithm

Very simple algorithm, there is an example here [coordinate ascent example](https://github.com/lnshi/ml-exercises/blob/master/ml_basics/rdm011_support_vector_machines/coordinate_ascent_example.png) to demonstrate how it works.

### SMO

For readabilities, i copy above our final dual problem (5.7) here:

$$
\begin{align*} \tag{5.7 copy}
  &\min _{\alpha} \Bigg[\frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n \alpha_i \alpha_j y_i y_j x_i^T x_j - \sum_{i=1}^n \alpha_i \Bigg] \\
  &s.t. \sum_{i=1}^n \alpha_{i} y_{i} = 0 \\
  &\quad \,\,\, \alpha_i \geq 0, \forall i
\end{align*}
$$

In coordinate ascent algorithm, after has done the random initialization(CANNOT initialize with 0), in each iteration we choose only one variable to update.

But with above problem, there is constraint $\sum\limits_{i=1}^n \alpha_{i} y_{i} = 0$, so we CANNOT just update one variable at a time, but at least update another one correspondingly to satisfy the constraint, so instead of one we choose two variables to tune in one iteration in SMO.

Detailed steps to show how to update the randomly chosen(actually there is some strategies to decide which better to chose) two variables $(\alpha_i, \alpha_j)$ in each iteration can be referred here: [SMO example](https://github.com/lnshi/ml-exercises/blob/master/ml_basics/rdm011_support_vector_machines/smo.png), full article from here: [机器学习算法实践-SVM中的SMO算法](https://zhuanlan.zhihu.com/p/29212107).

## Slack variables and penalty factors

For readabilities I just copy above formula (4) - the original problem here:

$$
\begin{align*} \tag{4 copy}
  &arg \, \underset{w, b}{min} \, \frac{1}{2}||w||^2 \\
  &s.t. \, y_i(w^Tx_i + b) \geq 1, \forall i
\end{align*}
$$

Introduce in slack variables $\zeta_i$ and penalty factor $C$:

$$
\begin{align*} \tag{4.1}
  &arg \, \underset{w, b}{min} \, \frac{1}{2}||w||^2 + C \sum_{i=1}^n \zeta_i \\
  &s.t. \, y_i(w^Tx_i + b) \geq 1 - \zeta_i, \forall i \\
  &\quad \,\,\, \zeta_i \geq 0, \forall i
\end{align*}
$$

For each sample point we have different slack variable(target solving the outliers, for the others the slack variables should be all 0), similarly for the penalty factor we also can have different values for each of the sample point, NOT necessarily to be all same.

Deal with unbalanced dataset(e.g.: the `+ class` has quite a lot of samples, but the `- class` only has quite a few):

$$
\begin{align*} \tag{4.2}
  &arg \, \underset{w, b}{min} \, \frac{1}{2}||w||^2 + C_+ \sum_{i=1}^p \zeta_i + C_- \sum_{i=p+1}^n \zeta_i \\
  &s.t. \, y_i(w^Tx_i + b) \geq 1 - \zeta_i, \forall i \\
  &\quad \,\,\, \zeta_i \geq 0, \forall i
\end{align*}
$$

# Experience sklearn built-in SVM

```python
import os

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Sets the backend of matplotlib to the 'inline' backend.
#
# With this backend, the output of plotting commands is displayed inline within frontends like the Jupyter notebook,
# directly below the code cell that produced it.
# The resulting plots will then also be stored in the notebook document.
#
# More details: https://stackoverflow.com/questions/43027980/purpose-of-matplotlib-inline
%matplotlib inline

from scipy.io import loadmat

raw_data = loadmat(os.getcwd() + '/linear_discriminable_samples.mat')
```

```python
data = pd.DataFrame(raw_data['X'], columns=['X1', 'X2'])
data['y'] = raw_data['y']

positive = data[data['y'].isin([1])]
negative = data[data['y'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['X1'], positive['X2'], s=50, marker='x', label='Positive')
ax.scatter(negative['X1'], negative['X2'], s=50, marker='o', label='Negative')
ax.legend()
```

## SVM with linear kernel ('C=1' vs 'C=100')

```python
from sklearn import svm

svc1 = svm.LinearSVC(C=1, loss='hinge', max_iter=1000)
svc1
```

    LinearSVC(C=1, class_weight=None, dual=True, fit_intercept=True,
         intercept_scaling=1, loss='hinge', max_iter=1000, multi_class='ovr',
         penalty='l2', random_state=None, tol=0.0001, verbose=0)

```python
svc1.fit(data[['X1', 'X2']], data['y'])
svc1.score(data[['X1', 'X2']], data['y'])
```

    /usr/local/anaconda3/lib/python3.7/site-packages/sklearn/svm/base.py:931: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      "the number of iterations.", ConvergenceWarning)

    0.9803921568627451

```python
svc2 = svm.LinearSVC(C=100, loss='hinge', max_iter=1000)
svc2.fit(data[['X1', 'X2']], data['y'])
svc2.score(data[['X1', 'X2']], data['y'])
```

    /usr/local/anaconda3/lib/python3.7/site-packages/sklearn/svm/base.py:931: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      "the number of iterations.", ConvergenceWarning)

    1.0

```python
data['SVM 1 Confidence'] = svc1.decision_function(data[['X1', 'X2']])

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(data['X1'], data['X2'], s=50, c=data['SVM 1 Confidence'], cmap='seismic')
ax.set_title('SVM (C=1) Decision Confidence')
```

```python
data['SVM 2 Confidence'] = svc2.decision_function(data[['X1', 'X2']])

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(data['X1'], data['X2'], s=50, c=data['SVM 2 Confidence'], cmap='seismic')
ax.set_title('SVM (C=100) Decision Confidence')
```

## SVM with gaussian kernel (built-in by sklearn already)

```python
raw_data = loadmat(os.getcwd() + '/nonlinear_discriminable_samples.mat')

data = pd.DataFrame(raw_data['X'], columns=['X1', 'X2'])
data['y'] = raw_data['y']

positive = data[data['y'].isin([1])]
negative = data[data['y'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['X1'], positive['X2'], s=30, marker='x', label='Positive')
ax.scatter(negative['X1'], negative['X2'], s=30, marker='o', label='Negative')
ax.legend()
```

### How to use a custom kernel with some dynamic values

```python
def build_gaussian_kernel(sigma):
    def gaussian_kernel(x1, x2):
        return np.exp(-(np.sum((x1 - x2) ** 2) / (2 * (sigma ** 2))))
    return gaussian_kernel
# Then: svc = svm.SVC(C=100, kernel=build_gaussian_kernel(sigma=1), gamma=10, probability=True)
```

```python
svc = svm.SVC(C=100, kernel='rbf', gamma=10, probability=True)
svc.fit(data[['X1', 'X2']], data['y'])
data['Probability'] = svc.predict_proba(data[['X1', 'X2']])[:,0]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(data['X1'], data['X2'], s=30, c=data['Probability'], cmap='Reds')
```

### Model selection: 'GridSearchCV' and 'PredefinedSplit'

```python
from sklearn import base
from sklearn import model_selection
from sklearn import ensemble

raw_data = loadmat(os.getcwd() + '/samples_contain_both_training_and_validation_set.mat')

X = np.concatenate(
    (raw_data['X'], raw_data['Xval']), axis=0
)
y = np.concatenate(
    (raw_data['y'].ravel(), raw_data['yval'].ravel())
)

test_fold = np.concatenate(
    (
        np.full(raw_data['X'].shape[0], 0),
        np.full(raw_data['Xval'].shape[0], -1)
    )
)
ps = model_selection.PredefinedSplit(test_fold)

param_candidates = [
    {
        'C': [0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100],
        'gamma': [0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100],
        'kernel': ['rbf']
    }
]

gs = model_selection.GridSearchCV(
    estimator=svm.SVC(),
    param_grid=param_candidates,
    cv=ps,
    refit=True,
    error_score=0,
    n_jobs=-1
)

gs.fit(X, y)
gs.best_estimator_, gs.best_score_
```

    (SVC(C=10, cache_size=200, class_weight=None, coef0=0.0,
       decision_function_shape='ovr', degree=3, gamma=10, kernel='rbf',
       max_iter=-1, probability=False, random_state=None, shrinking=True,
       tol=0.001, verbose=False), 0.943127962085308)

# References

- [支持向量机(SVM)是什么意思？](https://www.zhihu.com/question/21094489/answer/117246987)

- [支持向量机中的函数距离和几何距离怎么理解？](https://www.zhihu.com/question/20466147/answer/28469993)

- [SVM推导过程](https://zhuanlan.zhihu.com/p/34811858)

- [学习SVM（三）理解SVM中的对偶问题](https://blog.csdn.net/chaipp0607/article/details/73849539)

- [零基础学SVM—Support Vector Machine(一)](https://zhuanlan.zhihu.com/p/24638007)

- [零基础学SVM-Support Vector Machine(二)](https://zhuanlan.zhihu.com/p/29865057)

- [简易解说拉格朗日对偶（Lagrange duality）](http://www.cnblogs.com/90zeng/p/Lagrange_duality.html)

- [浅谈最优化问题的KKT条件](https://zhuanlan.zhihu.com/p/26514613)

- [机器学习算法实践-SVM中的SMO算法](https://zhuanlan.zhihu.com/p/29212107)

- [SVM学习（五）：松弛变量与惩罚因子](https://blog.csdn.net/qll125596718/article/details/6910921)


```python
import sympy as sp
import numpy as np
```

```python
def factorial(n):
    if n < 2:
        return 1
    else:
        return n*factorial(n-1)
```

```python
def n_degree_taylor(formula, x, x_0, n):
    '''
    Given the analytic expression, x and x_0, calculate
    its nth degree Taylor polynomial
    Note: Variable is defaulted to x
    '''
    var_x = sp.symbols('x')
    acc = 0
    # Be careful, n+1 because it's up to n
    for i in range(n+1):
        acc += sp.diff(formula, var_x, i).subs(var_x, x_0)/factorial(i)*(x-x_0)**i
    return float(acc)
```

```python
# Generic Test for n_degree_taylor
expected = -2
actual = n_degree_taylor('-sin(x)', 2, 0, 2)
tol = 10**-7
assert abs(expected-actual) < tol
```

```python
def empirical_taylor(derivatives, x, x_0):
    '''
    Given empirical derivatives f(x_0), f'(x_0), f''(x_0)...
    calculate it's taylor series with this empirical data
    ASSUME FROM 0th derivative (which is the original function)
    '''
    result = 0
    for i in range(len(derivatives)):
        result += derivatives[i]*(x-x_0)**i/factorial(i)
    return result
```

```python
# Generic Test Case for empirical_taylor
expected = -4/3
actual = empirical_taylor([8, -8, 4, -2, 2], 2, 0)
tol = 10**-7
assert abs(expected-actual) < tol
```

```python
# Workspace
n_degree_taylor('-sin(x)', 5, 0, 2)
```

    -5.0

```python

```


```python
import sympy
from sympy import I, conjugate, pi, oo
from sympy.functions import exp
from sympy.matrices import Identity, MatrixSymbol
import numpy as np
```

# \#3

## Fourier matrix with $\omega$

```python
ω = sympy.Symbol("omega")

def omegaPow(idx):
    return ω**(idx)

def fourierGenerator(N):
    return sympy.Matrix(N, N, lambda m, n: omegaPow(m * n))

assert fourierGenerator(2) == sympy.Matrix([[1, 1], [1, ω]])

fourierGenerator(4)
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & \omega & \omega^{2} & \omega^{3}\\1 & \omega^{2} & \omega^{4} & \omega^{6}\\1 & \omega^{3} & \omega^{6} & \omega^{9}\end{matrix}\right]$

```python
fourierGenerator(4)
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & \omega & \omega^{2} & \omega^{3}\\1 & \omega^{2} & \omega^{4} & \omega^{6}\\1 & \omega^{3} & \omega^{6} & \omega^{9}\end{matrix}\right]$

## Complex conjugate F

```python
conjugate(fourierGenerator(4))
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & \overline{\omega} & \overline{\omega}^{2} & \overline{\omega}^{3}\\1 & \overline{\omega}^{2} & \overline{\omega}^{4} & \overline{\omega}^{6}\\1 & \overline{\omega}^{3} & \overline{\omega}^{6} & \overline{\omega}^{9}\end{matrix}\right]$

## Substitute $\omega=e^\frac{i2{\pi}mn}{N}$

```python
def omegaSubstirution(idx, N):
    return sympy.exp((I * 2 * pi * idx) / N)

assert omegaSubstirution(1, 4) == I
assert omegaSubstirution(8, 8) == 1
assert omegaSubstirution(3, 6) == -1
```

## Generate Fourier matrix with values

```python
def fourierGeneratorWithExp(N):
    return sympy.Matrix(N, N, lambda m, n: omegaSubstirution(m * n, N))

assert fourierGeneratorWithExp(2) == sympy.Matrix([[1, 1], [1, -1]])

F4 = fourierGeneratorWithExp(4)

F4
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & i & -1 & - i\\1 & -1 & 1 & -1\\1 & - i & -1 & i\end{matrix}\right]$

## Matrix conjugate

```python
F4Conj = conjugate(F4)

assert Identity(4).as_explicit() == (1 / 4) * F4 * F4Conj

F4Conj
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & - i & -1 & i\\1 & -1 & 1 & -1\\1 & i & -1 & - i\end{matrix}\right]$

## Conjugate generator with $\omega$

```python
def fourierConjGenerator(N):
    return sympy.Matrix(N, N, lambda m, n: omegaPow(0 if m == 0 else (N - m) * n))

fourierConjGenerator(4)
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & \omega^{3} & \omega^{6} & \omega^{9}\\1 & \omega^{2} & \omega^{4} & \omega^{6}\\1 & \omega & \omega^{2} & \omega^{3}\end{matrix}\right]$

## Conjugate generator with values

```python
def fourierConjGeneratorWithExp(N):
    return sympy.Matrix(
        N, N, lambda m, n: omegaSubstirution(0 if m == 0 else (N - m) * n, N))

F4ConjWithExp = fourierConjGeneratorWithExp(4)

assert F4Conj == F4ConjWithExp

F4ConjWithExp
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & - i & -1 & i\\1 & -1 & 1 & -1\\1 & i & -1 & - i\end{matrix}\right]$

## Permutation Generator

```python
def generatePermutationMatrix(N):
    return np.vstack((np.hstack((np.array([1]), np.zeros(
        N - 1, dtype=int))), np.zeros((N - 1, N), dtype=int))) + np.fliplr(
            np.diagflat(np.ones(N - 1, dtype=int), -1))

assert np.all(
    generatePermutationMatrix(4) == np.array([[1, 0, 0, 0], [0, 0, 0, 1],
                                              [0, 0, 1, 0], [0, 1, 0, 0]]))

generatePermutationMatrix(4)
```

    array([[1, 0, 0, 0],
           [0, 0, 0, 1],
           [0, 0, 1, 0],
           [0, 1, 0, 0]])

## $$F=P{\cdot}\overline{F}$$

```python
P4 = generatePermutationMatrix(4)

assert F4 == P4 * F4Conj

P4 * F4Conj
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & i & -1 & - i\\1 & -1 & 1 & -1\\1 & - i & -1 & i\end{matrix}\right]$

## $$P^2 = I$$

```python
assert np.all(np.linalg.matrix_power(P4, 2) == np.identity(4, dtype=int))
```

## $$\frac{1}{N}{\cdot}F^2 = P$$

```python
assert np.all(((1 / 4) * np.linalg.matrix_power(F4, 2)).astype(int) == P4)
```

# \#4

```python
ID = sympy.Matrix(
    np.hstack((np.vstack((np.identity(2, dtype=int), np.identity(2,
                                                                 dtype=int))),
               np.vstack((np.identity(2, dtype=int) * np.diag(F4)[0:2],
                          -np.identity(2, dtype=int) * np.diag(F4)[0:2])))))

ID
```

$\displaystyle \left[\begin{matrix}1 & 0 & 1 & 0\\0 & 1 & 0 & i\\1 & 0 & -1 & 0\\0 & 1 & 0 & - i\end{matrix}\right]$

```python
ID.inv()
```

$\displaystyle \left[\begin{matrix}\frac{1}{2} & 0 & \frac{1}{2} & 0\\0 & \frac{1}{2} & 0 & \frac{1}{2}\\\frac{1}{2} & 0 & - \frac{1}{2} & 0\\0 & - \frac{i}{2} & 0 & \frac{i}{2}\end{matrix}\right]$

```python
F2 = fourierGeneratorWithExp(2)
Zeros2 = np.zeros((2, 2), dtype=int)

F22 = sympy.Matrix(
    np.array(np.vstack((np.hstack((F2, Zeros2)), np.hstack((Zeros2, F2))))))

F22
```

$\displaystyle \left[\begin{matrix}1 & 1 & 0 & 0\\1 & -1 & 0 & 0\\0 & 0 & 1 & 1\\0 & 0 & 1 & -1\end{matrix}\right]$

```python
F22.inv()
```

$\displaystyle \left[\begin{matrix}\frac{1}{2} & \frac{1}{2} & 0 & 0\\\frac{1}{2} & - \frac{1}{2} & 0 & 0\\0 & 0 & \frac{1}{2} & \frac{1}{2}\\0 & 0 & \frac{1}{2} & - \frac{1}{2}\end{matrix}\right]$

```python
EvenOddP = sympy.Matrix([[1, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0],
                         [0, 0, 0, 1]])
EvenOddP
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 0 & 1 & 0\\0 & 1 & 0 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
assert ID * F22 * EvenOddP == F4

ID * F22 * EvenOddP
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & i & -1 & - i\\1 & -1 & 1 & -1\\1 & - i & -1 & i\end{matrix}\right]$

```python
F4
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & i & -1 & - i\\1 & -1 & 1 & -1\\1 & - i & -1 & i\end{matrix}\right]$

# \#5

$$
 F^{T}_{N} =
\left(
    \begin{bmatrix}
       I_{N/2} & D_{N/2}  \\[0.3em]
       I_{N/2} & -D_{N/2} \\[0.3em]
     \end{bmatrix}
     \cdot
     \begin{bmatrix}
       F_{N/2} &   \\[0.3em]
        & F_{N/2} \\[0.3em]
     \end{bmatrix}
     \cdot
     \begin{bmatrix}
       even_{N/2} & even_{N/2}  \\[0.3em]
       odd_{N/2} & odd_{N/2} \\[0.3em]
     \end{bmatrix}
 \right)^T
     =
    \begin{bmatrix}
       even_{N/2} & odd_{N/2}  \\[0.3em]
       even_{N/2} & odd_{N/2} \\[0.3em]
     \end{bmatrix}
    \cdot
    \begin{bmatrix}
        F_{N/2} &   \\[0.3em]
        & F_{N/2} \\[0.3em]
    \end{bmatrix}
    \cdot
    \begin{bmatrix}
        I_{N/2} & I_{N/2} \\[0.3em]
        D_{N/2} & -D_{N/2} \\[0.3em]
    \end{bmatrix}
 $$

```python
EvenOddP * F22 * sympy.Matrix.transpose(ID)
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1\\1 & i & -1 & - i\\1 & -1 & 1 & -1\\1 & - i & -1 & i\end{matrix}\right]$

```python
EvenOddP * F22 * sympy.Matrix.transpose(ID) == F4
```

    True

# \#6

### Based on Permutation Matrix for the Conjugate Matrix - even / odd Permutation

```python
N = 6

P = generatePermutationMatrix(N)

EvenOddP = sympy.Matrix(
    np.vstack((
        P[0],
        P[np.arange(N-2, 1, -2)],
        P[np.arange(N-1, 0, -2)]
    ))
)

EvenOddP
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 1\end{matrix}\right]$

```python
def generatePermutationFourierMatrix(N):
    P = generatePermutationMatrix(N)
    return sympy.Matrix(
        np.vstack((
            P[0],
            P[np.arange(N-2, 1, -2)],
            P[np.arange(N-1, 0, -2)]
        ))
    )

assert np.all(generatePermutationFourierMatrix(4) == sympy.Matrix([
    [1, 0, 0, 0],
    [0, 0, 1, 0],
    [0, 1, 0, 0],
    [0, 0, 0, 1]
]))
```

### Shape of the Permutation Matrix
$$
\begin{bmatrix}
    even_{N/2} & even_{N/2}  \\[0.3em]
    odd_{N/2} & odd_{N/2} \\[0.3em]
\end{bmatrix}
$$

### N = 8 Fourier Permutation Matrix

```python
generatePermutationFourierMatrix(8)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 1 & 0\\0 & 1 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\end{matrix}\right]$

```python
generatePermutationFourierMatrix(6)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 1\end{matrix}\right]$

## Generate Fourier Matrices

### Generate Fourier Blocks for basic Matrix:

#### Half Eye: $I_{N/2}$
#### Half Fourier Diagonal: $D_{N/2}$
#### Half Fourier: $F_{N/2}$

```python
def generateFourierBlocks(N, fourierGenerator):
    half = int(N/2)
    quarterEye = sympy.Matrix.eye(half, half)
    quarterZeroes = sympy.Matrix.zeros(half, half)
    halfDiagFourier = sympy.Matrix.diag(
        np.diagonal(fourierGenerator(N))
    )[:half, :half]
    halfFourier = fourierGenerator(half)
    return (quarterEye, quarterZeroes, halfDiagFourier, halfFourier)
```

```python
Blocks4 = generateFourierBlocks(4, fourierGenerator)

assert Blocks4 == (
    sympy.Matrix([
        [1, 0],
        [0, 1]
    ]),
    sympy.Matrix([
        [0, 0],
        [0, 0]
    ]),
    sympy.Matrix([
        [1, 0],
        [0, ω]
    ]),
    sympy.Matrix([
        [1, 1],
        [1, ω]
    ])
)
```

```python
Blocks4Complex = generateFourierBlocks(4, fourierGeneratorWithExp)

assert Blocks4Complex == (
    sympy.Matrix([
        [1, 0],
        [0, 1]
    ]),
    sympy.Matrix([
        [0, 0],
        [0, 0]
    ]),
    sympy.Matrix([
        [1, 0],
        [0, I]
    ]),
    sympy.Matrix([
        [1, 1],
        [1, -1]
    ])
)
```

```python
def composeFourierMatricesFromBlocks(N, quarterEye, quarterZeroes, halfDiagFourier, halfFourier):
    return (
        sympy.Matrix.vstack(
            sympy.Matrix.hstack(
                quarterEye,
                halfDiagFourier
            ),
            sympy.Matrix.hstack(
                quarterEye,
                -halfDiagFourier
            )
        ),
        sympy.Matrix.vstack(
            sympy.Matrix.hstack(
                halfFourier,
                quarterZeroes
            ),
            sympy.Matrix.hstack(
                quarterZeroes,
                halfFourier
            )
        ),
        generatePermutationFourierMatrix(N)
    )
```

```python
def createFourierMatrices(N, fourierGenerator):
    (quarterEye, quarterZeroes, halfDiagFourier, halfFourier) = generateFourierBlocks(N, fourierGenerator)
    return composeFourierMatricesFromBlocks(N, quarterEye, quarterZeroes, halfDiagFourier, halfFourier)
```

## Generate Fourier Matrices

### IdentityAndDiagonal: $
\begin{bmatrix}
   I_{N/2} & D_{N/2}  \\[0.3em]
   I_{N/2} & -D_{N/2} \\[0.3em]
 \end{bmatrix}
$

### FourierHalfNSize: $
\begin{bmatrix}
   F_{N/2} &   \\[0.3em]
    & F_{N/2} \\[0.3em]
 \end{bmatrix}
$

### EvenOddPermutation: $
\begin{bmatrix}
    even_{N/2} & even_{N/2}  \\[0.3em]
    odd_{N/2} & odd_{N/2} \\[0.3em]
\end{bmatrix}
$

### Full picture
$$
F_{N}=\begin{bmatrix}
   I_{N/2} & D_{N/2}  \\[0.3em]
   I_{N/2} & -D_{N/2} \\[0.3em]
 \end{bmatrix}
 \cdot
 \begin{bmatrix}
   F_{N/2} &   \\[0.3em]
    & F_{N/2} \\[0.3em]
 \end{bmatrix}
 \cdot
 \begin{bmatrix}
   even_{N/2} & even_{N/2}  \\[0.3em]
   odd_{N/2} & odd_{N/2} \\[0.3em]
 \end{bmatrix}
 $$

```python
def generateFourierMatricesWithOmega(N):
    return createFourierMatrices(N, fourierGenerator)
```

```python
IdentityAndDiagonal, FHalfNSize, EvenOddPermutation = generateFourierMatricesWithOmega(4)
```

```python
assert sympy.Matrix([
    [1, 0, 1, 0],
    [0, 1, 0, ω],
    [1, 0, -1, 0],
    [0, 1, 0, -ω],
]) == IdentityAndDiagonal

IdentityAndDiagonal
```

$\displaystyle \left[\begin{matrix}1 & 0 & 1 & 0\\0 & 1 & 0 & \omega\\1 & 0 & -1 & 0\\0 & 1 & 0 & - \omega\end{matrix}\right]$

```python
assert sympy.Matrix([
    [1, 1, 0, 0],
    [1, ω, 0, 0],
    [0, 0, 1, 1],
    [0, 0, 1, ω]
]) == FHalfNSize

FHalfNSize
```

$\displaystyle \left[\begin{matrix}1 & 1 & 0 & 0\\1 & \omega & 0 & 0\\0 & 0 & 1 & 1\\0 & 0 & 1 & \omega\end{matrix}\right]$

```python
FHalfNSize = sympy
```

```python
assert sympy.Matrix([
    [1, 0, 0, 0],
    [0, 0, 1, 0],
    [0, 1, 0, 0],
    [0, 0, 0, 1]
]) == EvenOddPermutation

EvenOddPermutation
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 0 & 1 & 0\\0 & 1 & 0 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
def generateFourierMatricesWithExp(N):
    return createFourierMatrices(N, fourierGeneratorWithExp)
```

```python
IdentityAndDiagonal, FourierHalfNSize, EvenOddPermutation = generateFourierMatricesWithExp(4)
```

```python
assert sympy.Matrix([
    [1, 0, 1, 0],
    [0, 1, 0, I],
    [1, 0, -1, 0],
    [0, 1, 0, -I]
]) == IdentityAndDiagonal

IdentityAndDiagonal
```

$\displaystyle \left[\begin{matrix}1 & 0 & 1 & 0\\0 & 1 & 0 & i\\1 & 0 & -1 & 0\\0 & 1 & 0 & - i\end{matrix}\right]$

```python
assert sympy.Matrix([
    [1, 1, 0, 0],
    [1, -1, 0, 0],
    [0, 0, 1, 1],
    [0, 0, 1, -1]
]) == FourierHalfNSize

FourierHalfNSize
```

$\displaystyle \left[\begin{matrix}1 & 1 & 0 & 0\\1 & -1 & 0 & 0\\0 & 0 & 1 & 1\\0 & 0 & 1 & -1\end{matrix}\right]$

```python
assert sympy.Matrix([
    [1, 0, 0, 0],
    [0, 0, 1, 0],
    [0, 1, 0, 0],
    [0, 0, 0, 1]
]) == EvenOddPermutation

EvenOddPermutation
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 0 & 1 & 0\\0 & 1 & 0 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

## Solution for \#6

```python
IdentityAndDiagonal = sympy.Matrix([
    [1, 0, 0,  1,    0,    0],
    [0, 1, 0,  0,    ω,    0],
    [0, 0, 1,  0,    0,    ω**2],
    [1, 0, 0, -1,    0,    0],
    [0, 1, 0,  0,   -ω,    0],
    [0, 0, 1,  0,    0,   -ω**2],
])
IdentityAndDiagonal
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 1 & 0 & 0\\0 & 1 & 0 & 0 & \omega & 0\\0 & 0 & 1 & 0 & 0 & \omega^{2}\\1 & 0 & 0 & -1 & 0 & 0\\0 & 1 & 0 & 0 & - \omega & 0\\0 & 0 & 1 & 0 & 0 & - \omega^{2}\end{matrix}\right]$

```python
FourierHalfNSize = sympy.Matrix([
    [1, 1,    1,    0,    0,    0],
    [1, ω**2, ω**4, 0,    0,    0],
    [1, ω**4, ω**2, 0,    0,    0],
    [0, 0,    0,    1,    1,    1],
    [0, 0,    0,    1,    ω**2, ω**4],
    [0, 0,    0,    1,    ω**4, ω**2],
])

FourierHalfNSize
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 0 & 0 & 0\\1 & \omega^{2} & \omega^{4} & 0 & 0 & 0\\1 & \omega^{4} & \omega^{2} & 0 & 0 & 0\\0 & 0 & 0 & 1 & 1 & 1\\0 & 0 & 0 & 1 & \omega^{2} & \omega^{4}\\0 & 0 & 0 & 1 & \omega^{4} & \omega^{2}\end{matrix}\right]$

```python
EvenOddPermutation
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 1\end{matrix}\right]$

```python
FourierBasedOnMatrices = IdentityAndDiagonal * FourierHalfNSize * EvenOddPermutation
FourierBasedOnMatrices
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1 & 1 & 1\\1 & \omega & \omega^{2} & \omega^{3} & \omega^{4} & \omega^{5}\\1 & \omega^{2} & \omega^{4} & \omega^{6} & \omega^{2} & \omega^{4}\\1 & -1 & 1 & -1 & 1 & -1\\1 & - \omega & \omega^{2} & - \omega^{3} & \omega^{4} & - \omega^{5}\\1 & - \omega^{2} & \omega^{4} & - \omega^{6} & \omega^{2} & - \omega^{4}\end{matrix}\right]$

```python
# assert FourierBasedOnMatrices == fourierGeneratorWithExp(N)

fourierGenerator(N)
```

$\displaystyle \left[\begin{matrix}1 & 1 & 1 & 1 & 1 & 1\\1 & \omega & \omega^{2} & \omega^{3} & \omega^{4} & \omega^{5}\\1 & \omega^{2} & \omega^{4} & \omega^{6} & \omega^{8} & \omega^{10}\\1 & \omega^{3} & \omega^{6} & \omega^{9} & \omega^{12} & \omega^{15}\\1 & \omega^{4} & \omega^{8} & \omega^{12} & \omega^{16} & \omega^{20}\\1 & \omega^{5} & \omega^{10} & \omega^{15} & \omega^{20} & \omega^{25}\end{matrix}\right]$

```python
N = 4
sympy.Matrix(
    sympy.fft(sympy.Matrix.eye(2, 2), 1)
)
```

$\displaystyle \left[\begin{matrix}2\\1.0 - 1.0 i\\0\\1.0 + 1.0 i\end{matrix}\right]$

```python
N = 16
half = int(N/2)

sympy.fft(sympy.Matrix.eye(2), 1)

```

    [2, 1.0 - 1.0*I, 0, 1.0 + 1.0*I]

```python
sympy.Matrix.eye(half)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 1 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 1 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 1 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 1 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\end{matrix}\right]$


## RIHAD VARIAWA, Data Scientist - Who has fun LEARNING, EXPLORING & GROWING
## Functions
So far in this course we've explored equations that perform algebraic operations to produce one or more results. A *function* is a way of encapsulating an operation that takes an input and produces exactly one ouput.

For example, consider the following function definition:

\begin{equation}f(x) = x^{2} + 2\end{equation}

This defines a function named ***f*** that accepts one input (***x***) and returns a single value that is the result calculated by the expression *x2 + 2*.

Having defined the function, we can use it for any input value. For example:

\begin{equation}f(3) = 11\end{equation}

You've already seen a few examples of Python functions, which are defined using the **def** keyword. However, the strict definition of an algebraic function is that it must return a single value. Here's an example of defining and using a Python function that meets this criteria:

```python
# define a function to return x^2 + 2
def f(x):
    return x**2 + 2

# call the function
f(3)
```

You can use functions in equations, just like any other term. For example, consider the following equation:

\begin{equation}y = f(x) - 1\end{equation}

To calculate a value for ***y***, we take the ***f*** of ***x*** and subtract 1. So assuming that ***f*** is defined as previously, given an ***x*** value of 4, this equation returns a ***y*** value of **17** (*f*(4) returns 42 + 2, so 16 + 2 = 18; and then the equation subtracts 1 to give us 17). Here it is in Python:

```python
x = 4
y = f(x) - 1
print(y)
```

Of course, the value returned by a function depends on the input; and you can graph this with the iput (let's call it ***x***) on one axis and the output (***f(x)***) on the other.

```python
%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = np.array(range(-100, 101))

# Set up the graph
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()

# Plot x against f(x)
plt.plot(x,f(x), color='purple')

plt.show()
```

As you can see (if you hadn't already figured it out), our function is a *quadratic function* - it returns a squared value that results in a parabolic graph when the output for multiple input values are plotted.

## Bounds of a Function
Some functions will work for any input and may return any output. For example, consider the function ***u*** defined here:

\begin{equation}u(x) = x + 1\end{equation}

This function simply adds 1 to whatever input is passed to it, so it will produce a defined output for any value of ***x*** that is a *real* number; in other words, any "regular" number - but not an *imaginary* number like √-1, or ∞ (infinity). You can specify the set of real numbers using the symbol ${\rm I\!R}$ (note the double stroke). The values that can be used for ***x*** can be expressed as a *set*, which we indicate by enclosing all of the members of the set in "{...}" braces; so to indicate the set of all possible values for x such that x is a member of the set of all real numbers, we can use the following expression:

\begin{equation}\{x \in \rm I\!R\}\end{equation}

### Domain of a Function
We call the set of numbers for which a function can return value it's *domain*, and in this case, the domain of ***u*** is the set of all real numbers; which is actually the default assumption for most functions.

Now consider the following function ***g***:

\begin{equation}g(x) = (\frac{12}{2x})^{2}\end{equation}

If we use this function with an ***x*** value of **2**, we would get the output **9**; because (12 ÷ (2•2))2 is 9. Similarly, if we use the value **-3** for ***x***, the output will be **4**. However, what happens when we apply this function to an ***x*** value of **0**? Anything divided by 0 is undefined, so the function ***g*** doesn't work for an ***x*** value of 0.

So we need a way to denote the domain of the function ***g*** by indicating the input values for which a defined output can be returned. Specifically, we need to restrict ***x*** to a specific list of values - specifically any real number that is not 0. To indicate this, we can use the following notation:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; x \ne 0 \}\end{equation}

This is interpreted as *Any value for x where x is in the set of real numbers such that x is not equal to 0*, and we can incorporate this into the function's definition like this:

\begin{equation}g(x) = (\frac{12}{2x})^{2}, \{x \in \rm I\!R\;\;|\;\; x \ne 0 \}\end{equation}

Or more simply:

\begin{equation}g(x) = (\frac{12}{2x})^{2},\;\; x \ne 0\end{equation}

When you plot the output of a function, you can indicate the gaps caused by input values that are not in the function's domain by plotting an empty circle to show that the function is not defined at this point:

```python
%matplotlib inline

# Define function g
def g(x):
    if x != 0:
        return (12/2*x)**2

# Plot output from function g
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = range(-100, 101)

# Get the corresponding y values from the function
y = [g(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('g(x)')
plt.grid()

# Plot x against g(x)
plt.plot(x,y, color='purple')

# plot an empty circle to show the undefined point
plt.plot(0,g(0.0000001), color='purple', marker='o', markerfacecolor='w', markersize=8)

plt.show()
```

Note that the function works for every value other than 0; so the function is defined for x = 0.000000001, and for x = -0.000000001; it only fails to return a defined value for exactly 0.

OK, let's take another example. Consider this function:

\begin{equation}h(x) = 2\sqrt{x}\end{equation}

Applying this function to a non-negative ***x*** value returns a meaningful output; but for any value where ***x*** is negative, the output is undefined.

We can indicate the domain of this function in its definition like this:

\begin{equation}h(x) = 2\sqrt{x}, \{x \in \rm I\!R\;\;|\;\; x \ge 0 \}\end{equation}

This is interpreted as *Any value for x where x is in the set of real numbers such that x is greater than or equal to 0*.

Or, you might see this in a simpler format:

\begin{equation}h(x) = 2\sqrt{x},\;\; x \ge 0\end{equation}

Note that the symbol ≥ is used to indicate that the value must be *greater than **or equal to*** 0; and this means that **0** is included in the set of valid values. To indicate that the value must be *greater than 0, **not including 0***, use the > symbol. You can also use the equivalent symbols for *less than or equal to* (≤) and *less than* (<).

When plotting a function line that marks the end of a continuous range, the end of the line is shown as a circle, which is filled if the function includes the value at that point, and unfilled if it does not.

Here's the Python to plot function ***h***:

```python
%matplotlib inline

def h(x):
    if x >= 0:
        import numpy as np
        return 2 * np.sqrt(x)

# Plot output from function h
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = range(-100, 101)

# Get the corresponding y values from the function
y = [h(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('h(x)')
plt.grid()

# Plot x against h(x)
plt.plot(x,y, color='purple')

# plot a filled circle at the end to indicate a closed interval
plt.plot(0, h(0), color='purple', marker='o', markerfacecolor='purple', markersize=8)

plt.show()
```

Sometimes, a function may be defined for a specific *interval*; for example, for all values between 0 and 5:

\begin{equation}j(x) = x + 2,\;\; x \ge 0 \text{ and } x \le 5\end{equation}

In this case, the function is defined for ***x*** values between 0 and 5 *inclusive*; in other words, **0** and **5** are included in the set of defined values. This is known as a *closed* interval and can be indicated like this:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; 0 \le x \le 5 \}\end{equation}

It could also be indicated like this:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; [0,5] \}\end{equation}

If the condition in the function was **x > 0 and x < 5**, then the interval would be described as *open* and 0 and 5 would *not* be included in the set of defined values. This would be indicated using one of the following expressions:

\begin{equation}\{x \in \rm I\!R\;\;|\;\; 0 \lt x \lt 5 \}\end{equation}
\begin{equation}\{x \in \rm I\!R\;\;|\;\; (0,5) \}\end{equation}

Here's function ***j*** in Python:

```python
%matplotlib inline

def j(x):
    if x >= 0 and x <= 5:
        return x + 2

# Plot output from function j
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = range(-100, 101)
y = [j(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('j(x)')
plt.grid()

# Plot x against k(x)
plt.plot(x, y, color='purple')

# plot a filled circle at the ends to indicate an open interval
plt.plot(0, j(0), color='purple', marker='o', markerfacecolor='purple', markersize=8)
plt.plot(5, j(5), color='purple', marker='o', markerfacecolor='purple', markersize=8)

plt.show()
```

Now, suppose we have a function like this:

\begin{equation}
k(x) = \begin{cases}
  0, & \text{if } x = 0, \\
  1, & \text{if } x = 100
\end{cases}
\end{equation}

In this case, the function has highly restricted domain; it only returns a defined output for 0 and 100. No output for any other ***x*** value is defined. In this case, the set of the domain is:

\begin{equation}\{0,100\}\end{equation}

Note that this does not include all real numbers, it only includes 0 and 100.

When we use Python to plot this function, note that it only makes sense to plot a scatter plot showing the individual values returned, there is no line in between because the function is not continuous between the values within the domain.

```python
%matplotlib inline

def k(x):
    if x == 0:
        return 0
    elif x == 100:
        return 1

# Plot output from function k
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = range(-100, 101)
# Get the k(x) values for every value in x
y = [k(a) for a in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('k(x)')
plt.grid()

# Plot x against k(x)
plt.scatter(x, y, color='purple')

plt.show()
```

### Range of a Function
Just as the domain of a function defines the set of values for which the function is defined, the *range* of a function defines the set of possible outputs from the function.

For example, consider the following function:

\begin{equation}p(x) = x^{2} + 1\end{equation}

The domain of this function is all real numbers. However, this is a quadratic function, so the output values will form a parabola; and since the function has no negative coefficient or constant, it will be an upward opening parabola with a vertex that has a y value of 1.

So what does that tell us? Well, the minimum value that will be returned by this function is 1, so it's range is:

\begin{equation}\{p(x) \in \rm I\!R\;\;|\;\; p(x) \ge 1 \}\end{equation}

Let's create and plot the function for a range of ***x*** values in Python:

```python
%matplotlib inline

# define a function to return x^2 + 1
def p(x):
    return x**2 + 1

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from -100 to 100
x = np.array(range(-100, 101))

# Set up the graph
plt.xlabel('x')
plt.ylabel('p(x)')
plt.grid()

# Plot x against f(x)
plt.plot(x,p(x), color='purple')

plt.show()
```

Note that the ***p(x)*** values in the plot drop exponentially for ***x*** values that are negative, and then rise exponentially for positive ***x*** values; but the minimum value returned by the function (for an *x* value of 0) is **1**.

AI Professional/2 - Essential Mathematics For Artificial Intelligence/DAT256x/Module01/01-08-Functions.ipynb


# Solving orbital equations with different algorithms

This notebook was adapted from `Orbit_games.ipynb`.

We consider energy plots and orbital solutions in polar coordinates for the general potential energy

$\begin{align}
   U(r) = k r^n
\end{align}$

for different ODE solution algorithms.  The `solve_ivp` function can itself be specified to use different solution methods (with the `method` keyword).  Here we will set it by default to use 'RK23', which is a variant on the Runge-Kutta second-order algorithm.  Second-order in this context means that the accuracy of a calculation will improve by a factor of $10^2 = 100$ if $\Delta t$ is reduced by a factor of ten.

We will compare it with the crudest algorithm, Euler's method, which is first order, and a second-order algorithm called Leapfrog, which is designed to be precisely time-reversal invariant.  This property guarantees conservation of energy, which is not true of the other algorithms we will consider.

To solve the differential equations for orbits, we have defined the $\mathbf{y}$
and $d\mathbf{y}/dt$ vectors as

$\begin{align}
  \mathbf{y} = \left(\begin{array}{c} r(t) \\ \dot r(t) \\ \phi(t)  \end{array} \right)
  \qquad
  \frac{d\mathbf{y}}{dt}
       = \left(\begin{array}{c} \dot r(t) \\ \ddot r(t) \\ \dot\phi(t) \end{array} \right)
       = \left(\begin{array}{c} \dot r(t) \\
                                 -\frac{1}{\mu}\frac{dU_{\rm eff}(r)}{dr} \\
                                 \frac{l}{\mu r^2} \end{array} \right)
\end{align}$

where we have substituted the differential equations for $\ddot r$ and $\dot\phi$.

Then Euler's method can be written as a simple prescription to obtain $\mathbf{y}_{i+1}$
from $\mathbf{y}_i$, where the subscripts label the elements of the `t_pts` array:
$\mathbf{y}_{i+1} = \mathbf{y}_i + \left(d\mathbf{y}/dt\right)_i \Delta t$, or, by components:

$\begin{align}
   r_{i+1} &= r_i + \frac{d\mathbf{y}_i[0]}{dt}  \Delta t  \\
   \dot r_{i+1} &= \dot r_{i} + \frac{d\mathbf{y}_i[1]}{dt}  \Delta t \\
   \phi_{i+1} &= \phi_i + \frac{d\mathbf{y}_i[2]}{dt} \Delta t
\end{align}$

**Look at the** `solve_ode_Euler` **method below and verify the algorithm is correctly implemented.**

The leapfrog method does better by evaluating $\dot r$ at a halfway time step before and after the $r$ evaluation,
which is both more accurate and incorporates time reversal:

$\begin{align}
   \dot r_{i+1/2} &= \dot r_{i} + \frac{d\mathbf{y}_i[1]}{dt}  \Delta t/2 \\
   r_{i+1} &= r_i +  \dot r_{i+1/2}  \Delta t  \\
   \dot r_{i+1} &= \dot r_{i+1/2} + \frac{d\mathbf{y}_{i+1}[1]}{dt}  \Delta t/2 \\
   \phi_{i+1} &= \phi_i + \frac{d\mathbf{y}_i[2]}{dt} \Delta t
\end{align}$

**Look at the** `solve_ode_Leapfrog` **method below and verify the algorithm is correctly implemented.**

A third method is the second-order Runge-Kutta algorithm, which we invoke from `solve_ivp` as `RK23`.
It does not use a fixed time-step as in our "homemade" implementations, so there is not a direct
comparison, but we can still check if it conserves energy.

**Run the notebook.  You are to turn in and comment on the "Change in energy with time" plot at the end.
Where do you see energy conserved or not conserved?  Show that Euler is first order and leapfrog is second
order by changing $\Delta t$; describe what you did and what you found.**

**Try another potential to see if you get the same general conclusions.**

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
```

```python
# Change the common font size
font_size = 14
plt.rcParams.update({'font.size': font_size})
```

```python
class Orbit:
    """
    Potentials and associated differential equations for central force motion
    with the potential U(r) = k r^n.  Several algorithms for integration of
    ordinary differential equations are now available.
    """

    def __init__(self, ang_mom, n, k=1, mu=1):
        self.ang_mom = ang_mom
        self.n = n
        self.k = k
        self.mu = mu

    def U(self, r):
        """Potential energy of the form U = kr^n."""
        return self.k * r**self.n

    def Ucf(self, r):
        """Centrifugal potential energy"""
        return self.ang_mom**2 / (2. * self.mu * r**2)

    def Ueff(self, r):
        """Effective potential energy"""
        return self.U(r) + self.Ucf(r)

    def U_deriv(self, r):
        """dU/dr"""
        return self.n * self.k * r**(self.n - 1)

    def Ucf_deriv(self, r):
        """dU_cf/dr"""
        return -2. * self.ang_mom**2 / (2. * self.mu * r**3)

    def Ueff_deriv(self, r):
        """dU_eff/dr"""
        return self.U_deriv(r) + self.Ucf_deriv(r)

    def dy_dt(self, t, y):
        """
        This function returns the right-hand side of the diffeq:
        [dr/dt d^2r/dt^2 dphi/dt]

        Parameters
        ----------
        t : float
            time
        y : float
            3-component vector with y[0] = r(t), y[1] = dr/dt, y[2] = phi

        """
        return [ y[1],
                -1./self.mu * self.Ueff_deriv(y[0]),
                self.ang_mom / (self.mu * y[0]**2) ]

    def solve_ode(self, t_pts, r_0, r_dot_0, phi_0,
                  method='RK23',
                  abserr=1.0e-8, relerr=1.0e-8):
        """
        Solve the ODE given initial conditions.
        Use solve_ivp with the option of specifying the method.
        Specify smaller abserr and relerr to get more precision.
        """
        y = [r_0, r_dot_0, phi_0]
        solution = solve_ivp(self.dy_dt, (t_pts[0], t_pts[-1]),
                             y, t_eval=t_pts, method=method,
                             atol=abserr, rtol=relerr)
        r, r_dot, phi = solution.y
        return r, r_dot, phi

    def solve_ode_Euler(self, t_pts, r_0, r_dot_0, phi_0):
        """
        Solve the ODE given initial conditions with the Euler method.
        The accuracy is determined by the spacing of times in t_pts.
        """

        delta_t = t_pts[1] - t_pts[0]

        # initialize the arrays for r, rdot, phi with zeros
        num_t_pts = len(t_pts)    # length of the array t_pts
        r = np.zeros(num_t_pts)
        r_dot = np.zeros(num_t_pts)
        phi = np.zeros(num_t_pts)

        # initial conditions
        r[0] = r_0
        r_dot[0] = r_dot_0
        phi[0] = phi_0

        # step through the differential equation
        for i in np.arange(num_t_pts - 1):
            t = t_pts[i]
            y = [r[i], r_dot[i], phi[i]]
            r[i+1] = r[i] + self.dy_dt(t,y)[0] * delta_t
            r_dot[i+1] = r_dot[i] + self.dy_dt(t,y)[1] * delta_t
            phi[i+1] = phi[i] + self.dy_dt(t,y)[2] * delta_t
        return r, r_dot, phi

    def solve_ode_Leapfrog(self, t_pts, r_0, r_dot_0, phi_0):
        """
        Solve the ODE given initial conditions with the Leapfrog method.
        """
        delta_t = t_pts[1] - t_pts[0]

        # initialize the arrays for r, rdot, r_dot_half, phi with zeros
        num_t_pts = len(t_pts)
        r = np.zeros(num_t_pts)
        r_dot = np.zeros(num_t_pts)
        r_dot_half = np.zeros(num_t_pts)
        phi = np.zeros(num_t_pts)

        # initial conditions
        r[0] = r_0
        r_dot[0] = r_dot_0
        phi[0] = phi_0

        # step through the differential equation
        for i in np.arange(num_t_pts - 1):
            t = t_pts[i]
            y = [r[i], r_dot[i], phi[i]]
            r_dot_half[i] = r_dot[i] + self.dy_dt(t, y)[1] * delta_t/2.
            r[i+1] = r[i] + r_dot_half[i] * delta_t

            y = [r[i+1], r_dot[i], phi[i]]
            r_dot[i+1] = r_dot_half[i] + self.dy_dt(t, y)[1] * delta_t/2.

            phi[i+1] = phi[i] + self.dy_dt(t,y)[2] * delta_t
        return r, r_dot, phi

    def energy(self, t_pts, r, r_dot):
        """Evaluate the energy as a function of time"""
        return (self.mu/2.) * r_dot**2 + self.Ueff(r)
```

```python
def start_stop_indices(t_pts, plot_start, plot_stop):
    start_index = (np.fabs(t_pts-plot_start)).argmin()  # index in t_pts array
    stop_index = (np.fabs(t_pts-plot_stop)).argmin()  # index in t_pts array
    return start_index, stop_index
```

# Pick a potential

```python
n = 2
k = 1.
ang_mom = 2.
o1 = Orbit(ang_mom, n=n, k=k, mu=1)

fig_2 = plt.figure(figsize=(7,5))
ax_2 = fig_2.add_subplot(1,1,1)

r_pts = np.linspace(0.001, 3., 200)
U_pts = o1.U(r_pts)
Ucf_pts = o1.Ucf(r_pts)
Ueff_pts = o1.Ueff(r_pts)

ax_2.plot(r_pts, U_pts, linestyle='dashed', color='blue', label='U(r)')
ax_2.plot(r_pts, Ucf_pts, linestyle='dotted', color='green', label='Ucf(r)')
ax_2.plot(r_pts, Ueff_pts, linestyle='solid', color='red', label='Ueff(r)')

ax_2.set_xlim(0., 3.)
ax_2.set_ylim(-1., 10.)
ax_2.set_xlabel('r')
ax_2.set_ylabel('U(r)')
ax_2.set_title(f'$n = {n},\ \ k = {k},\ \  l = {ang_mom}$')
ax_2.legend(loc='upper center')

ax_2.axhline(0., color='black', alpha=0.3)

fig_2.tight_layout()

fig_2.savefig('Gravitation_orbit_1.png')

```

## Plot orbit and check energy conservation

```python
# Plotting time
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)

# Initial conditions
r_0 = 1.  #  1.
r_dot_0 = 0.
phi_0 = 0.0
r_pts, r_dot_pts, phi_pts = o1.solve_ode(t_pts, r_0, r_dot_0, phi_0)
r_pts_Euler, r_dot_pts_Euler, phi_pts_Euler \
                              = o1.solve_ode_Euler(t_pts, r_0, r_dot_0, phi_0)
r_pts_LF, r_dot_pts_LF, phi_pts_LF \
                           = o1.solve_ode_Leapfrog(t_pts, r_0, r_dot_0, phi_0)

c = o1.ang_mom**2 / (np.abs(o1.k) * o1.mu)
epsilon = c / r_0 - 1.
energy_0 = o1.mu/2. * r_dot_0**2 + o1.Ueff(r_0)
print(f'energy = {energy_0:.2f}')
print(f'eccentricity = {epsilon:.2f}')
```

    energy = 3.00
    eccentricity = 3.00

```python
fig_4 = plt.figure(figsize=(8,8))

overall_title = 'Orbit:  ' + \
                rf' $n = {o1.n},$' + \
                rf' $k = {o1.k:.1f},$' + \
                rf' $l = {o1.ang_mom:.1f},$' + \
                rf' $r_0 = {r_0:.1f},$' + \
                rf' $\dot r_0 = {r_dot_0:.1f},$' + \
                rf' $\phi_0 = {phi_0:.1f}$' + \
                '\n'     # \n means a new line (adds some space here)
fig_4.suptitle(overall_title, va='baseline')

ax_4a = fig_4.add_subplot(2,2,1)
ax_4a.plot(t_pts, r_pts, color='black', label='RK23')
ax_4a.plot(t_pts, r_pts_Euler, color='blue', label='Euler')
ax_4a.plot(t_pts, r_pts_LF, color='red', label='Leapfrog')
ax_4a.set_xlabel(r'$t$')
ax_4a.set_ylabel(r'$r$')
ax_4a.set_title('Time dependence of radius')
ax_4a.legend()

ax_4b = fig_4.add_subplot(2,2,2)
ax_4b.plot(t_pts, phi_pts/(2.*np.pi), color='black', label='RK23')
ax_4b.plot(t_pts, phi_pts_Euler/(2.*np.pi), color='blue', label='Euler')
ax_4b.plot(t_pts, phi_pts_LF/(2.*np.pi), color='red', label='Leapfrog')
ax_4b.set_xlabel(r'$t$')
ax_4b.set_ylabel(r'$\phi/2\pi$')
ax_4b.set_title(r'Time dependence of $\phi$')
ax_4b.legend()

ax_4c = fig_4.add_subplot(2,2,3)
ax_4c.plot(r_pts*np.cos(phi_pts), r_pts*np.sin(phi_pts),
           color='black', label='RK23')
ax_4c.plot(r_pts_Euler*np.cos(phi_pts_Euler),
           r_pts_Euler*np.sin(phi_pts_Euler),
           color='blue', label='Euler')
ax_4c.plot(r_pts_LF*np.cos(phi_pts_LF),
           r_pts_LF*np.sin(phi_pts_LF),
           color='red', label='Leapfrog')
ax_4c.set_xlabel(r'$x$')
ax_4c.set_ylabel(r'$y$')
ax_4c.set_aspect('equal')
ax_4c.set_title('Cartesian plot')
ax_4c.legend()

ax_4d = fig_4.add_subplot(2,2,4, polar=True)
ax_4d.plot(phi_pts, r_pts, color='black', label='RK23')
ax_4d.plot(phi_pts_Euler, r_pts_Euler, color='blue', label='Euler')
ax_4d.plot(phi_pts_LF, r_pts_LF, color='red', label='Leapfrog')
ax_4d.set_title('Polar plot', pad=20.)
ax_4d.legend()

fig_4.tight_layout()
fig_4.savefig('Leapfrog_orbit_1.png', dpi=200, bbox_inches='tight')

```

```python
E_tot_pts = o1.energy(t_pts, r_pts, r_dot_pts)
E_tot_0 = E_tot_pts[0]
E_tot_rel_pts = np.abs((E_tot_pts - E_tot_0)/E_tot_0)

E_tot_pts_Euler = o1.energy(t_pts, r_pts_Euler, r_dot_pts_Euler)
E_tot_0_Euler = E_tot_pts_Euler[0]
E_tot_rel_pts_Euler = np.abs((E_tot_pts_Euler - E_tot_0_Euler)/E_tot_0_Euler)

E_tot_pts_LF = o1.energy(t_pts, r_pts_LF, r_dot_pts_LF)
E_tot_0_LF = E_tot_pts_LF[0]
E_tot_rel_pts_LF = np.abs((E_tot_pts_LF - E_tot_0_LF)/E_tot_0_LF)

```

```python
fig_5 = plt.figure(figsize=(6,6))

overall_title = 'Orbit:  ' + \
                rf' $n = {o1.n},$' + \
                rf' $k = {o1.k:.1f},$' + \
                rf' $l = {o1.ang_mom:.1f},$' + \
                rf' $r_0 = {r_0:.1f},$' + \
                rf' $\dot r_0 = {r_dot_0:.1f},$' + \
                rf' $\phi_0 = {phi_0:.1f}$' + \
                '\n'     # \n means a new line (adds some space here)
fig_5.suptitle(overall_title, va='baseline')

ax_5a = fig_5.add_subplot(1,1,1)
#ax_5a.semilogy(t_pts, np.abs(E_tot_pts), color='black', label=r'$E(t)$')
ax_5a.semilogy(t_pts, E_tot_rel_pts,
               color='green', label=r'$\Delta E(t)$ RK23')
ax_5a.semilogy(t_pts, E_tot_rel_pts_Euler,
               color='blue', label=r'$\Delta E(t)$ Euler')
ax_5a.semilogy(t_pts, E_tot_rel_pts_LF,
               color='red', label=r'$\Delta E(t)$ Leapfrog')
ax_5a.set_ylim(1.e-10, 1.e-2)    # (1.e-12, 5)
ax_5a.set_xlabel(r'$t$')
ax_5a.set_ylabel(r'Energy')
ax_5a.set_title('Change in energy with time')
ax_5a.legend()

fig_5.tight_layout()
fig_5.savefig('Leapfrog_energy_test_1.png', dpi=200, bbox_inches='tight')

```

```python

```

```python

```


```python
import numpy as np
from sympy import *
init_printing(use_unicode=True) # 輸出數學格式
import matplotlib.pylab as plt
%matplotlib
```

    Using matplotlib backend: Qt5Agg

```python
M = MatrixSymbol('M', 3, 3)
```

```python
Matrix(M)
```

$$\left[\begin{array}{ccc}M_{0, 0} & M_{0, 1} & M_{0, 2}\\M_{1, 0} & M_{1, 1} & M_{1, 2}\\M_{2, 0} & M_{2, 1} & M_{2, 2}\end{array}\right]$$

```python
x1, x2 = symbols('x1, x2')
w111, w112, w121, w122 = symbols('w111, w121, w112, w122')
w211, w212, w221, w222 = symbols('w211, w221, w212, w222')
b11, b12, b21, b22 = symbols('b11, b12, b21, b22')
X = Matrix([x1, x2]).T
W1 = Matrix([[w111, w121], [w112, w122]])
W2 = Matrix([[w211, w221], [w212, w222]])
B1 = Matrix([b11, b12]).T
B2 = Matrix([b21, b22]).T
```

```python
X * W1 + B1
```

$$\left[\begin{matrix}b_{11} + w_{111} x_{1} + w_{121} x_{2} & b_{12} + w_{112} x_{1} + w_{122} x_{2}\end{matrix}\right]$$

```python
(((X * W1) + B1) * W2) + B2
```

$$\left[\begin{matrix}b_{21} + w_{211} \left(b_{11} + w_{111} x_{1} + w_{121} x_{2}\right) + w_{221} \left(b_{12} + w_{112} x_{1} + w_{122} x_{2}\right) & b_{22} + w_{212} \left(b_{11} + w_{111} x_{1} + w_{121} x_{2}\right) + w_{222} \left(b_{12} + w_{112} x_{1} + w_{122} x_{2}\right)\end{matrix}\right]$$