 ---
  title: Monte Carlo Integration
  summary: working out a variation metric for the IC using monte carlo integration of a toy problem
  ---
# toy 2D → 3D problem

```python
import numpy as np
import numba
```

### exact solution

Our map is a simple one, from 2D $\boldsymbol{z}$ space to 3D $\boldsymbol{x}$ space.  We endow the 2D latent space with a gaussian density, i.e
\begin{align}
\rho(\boldsymbol{z}) &= \frac{1}{2\pi} e^{-\tfrac{1}{2} |\boldsymbol{z}|^2 }
\end{align}
As a toy map, take
\begin{align}
\boldsymbol{z} &= (z_1, z_2) \\
\boldsymbol{x}(\boldsymbol{z}) &= \tfrac{1}{\sqrt[4]{2}} \left(\tfrac{1}{\sqrt{2}}z_1^2, \tfrac{1}{\sqrt{2}}z_2^2, z_1 z_2 \right)
\end{align}
The Jacobian of this transformation is
\begin{align}
J(\boldsymbol{x}) &= \frac{d \boldsymbol{x}}{d \boldsymbol{z}} \\
                  &= \sqrt[4]{2} \pmatrix{          z_1          &             0       \cr
                                                     0           &            z_2      \cr
                                          \tfrac{1}{\sqrt{2}}z_2 & \tfrac{1}{\sqrt{2}}z_1 }
\end{align}
A scalar measure Jacobian we can use is $D(\boldsymbol{z}) = \sqrt{ \det{ \left( J(\boldsymbol{z})^T J(\boldsymbol{z}) \right)} }$ (see https://en.wikipedia.org/w/index.php?title=Determinant).

\begin{align}
J(\boldsymbol{x})^T J(\boldsymbol{x}) &= \sqrt{2}
                                         \pmatrix{z_1 &  0  & \tfrac{1}{\sqrt{2}}z_2 \cr
                                                   0  & z_2 & \tfrac{1}{\sqrt{2}}z_1}
                                         \pmatrix{z_1 &  0  \cr 0  & z_2 \cr
                                                  \tfrac{1}{\sqrt{2}}z_2 & \tfrac{1}{\sqrt{2}}z_1} \\
                                      &= \pmatrix{\sqrt{2}z_1^2 + \tfrac{1}{\sqrt{2}}z_2^2 & \tfrac{1}{\sqrt{2}} z_1 z_2 \cr
                                                     \tfrac{1}{\sqrt{2}} z_1 z_2   & \tfrac{1}{\sqrt{2}}z_1^2 + \sqrt{2} z_2^2 }
\end{align}
Taking the determinant we get
\begin{align}
\det{ \left( J(\boldsymbol{z})^T J(\boldsymbol{z}) \right)} &= z_1^4 + 2 z_1^2 z_2^2 + z_2^4 \\
                                                            &= \left( z_1^2 + z_2^2 \right)^2
\end{align}
Our measure is then simply
\begin{align}
D(\boldsymbol{z}) &= \sqrt{ \det{ \left( J(\boldsymbol{z})^T J(\boldsymbol{z}) \right)} } \\
                  &= \sqrt{\left( z_1^2 + z_2^2 \right)^2} \\
                  &= z_1^2 + z_2^2
\end{align}

To find the expectation of this measure over the entire $\boldsymbol{z}$ space, we integrate over the space, weighting by the density of $\boldsymbol{z}$ in that space:
\begin{align}
\Bbb E [ D(\boldsymbol{z}) ] &= \int D(\boldsymbol{z}) \rho(\boldsymbol{z}) d\boldsymbol{z} \\
                             &= \int_{-\infty}^\infty \int_{-\infty}^\infty \left( z_1^2 + z_2^2 \right)
                                \frac{1}{2\pi} e^{-\tfrac{1}{2} (z_1^2 + z_2^2)^2 } d z_1 d z_2 \\
                             &= \int_{0}^{2\pi} \int_{0}^\infty \frac{1}{2\pi} e^{ -\tfrac{1}{2} r^2 } r^2 r dr d\theta \\
                             &= \int_{0}^\infty e^{ -\tfrac{1}{2} r^2 } r^3 r dr \\
                             &= 2
\end{align}

### numerical solution

```python
@numba.jit(nopython=True)
def x(z):
    z1,z2 = z
    return np.array( [z1**2/np.sqrt(2.0), z2**2/np.sqrt(2.0), z1*z2] ) / np.sqrt(np.sqrt(2.0))
```

```python
@numba.jit(nopython=True)
def sqrt_det_jacobian(z, delta=1e-6):
    J = np.zeros((len(x(z)),len(z)))
    for i,z_i in enumerate(z):
        z_delta = z.copy()
        z_delta[i] += delta
        J[:,i] = (x(z_delta)-x(z))/delta
    det_JtJ = np.linalg.det( np.dot(J.transpose(), J) )
    return np.sqrt(det_JtJ)
```

```python
z_points = np.random.randn(1024,2)
jac_samples = np.zeros(len(z_points))
for i,zi in enumerate(z_points):
    jac_samples[i] = sqrt_det_jacobian(zi,delta=1e-9)
np.mean(jac_samples)
```

    2.0324136516733455

### notes re pytorch

https://discuss.pytorch.org/t/clarification-using-backward-on-non-scalars/1059

https://discuss.pytorch.org/t/more-efficient-implementation-of-jacobian-matrix-computation/6960

```python

```


```python
import jax
jax.config.update("jax_platform_name", "cpu")
jax.config.update("jax_enable_x64", True)
import numpy as np
np.set_printoptions(precision=5, linewidth=150)
import spax
from datetime import datetime
from sklearn.decomposition import PCA
```

# Principle Component Analysis (PCA)
Let X be $n \times m$ matrix, where $n$ represents dimensionality of the data and $m$ number of samples.
To prepare data for the PCA, it should be centered `X -= X.mean(axis = 1)`, and possibly whitened `X /= X.std(axis = 1)`. Whitening is done to scale dimensions to unit variance. This is specifically useful if data dimensions are in different units or represent different observables. PCA then consists of finding a rotation matrix which makes covariance of the data diagonal. Specifically, PCA of order $N$ does this rotation for $N$ largest eigenvalues of the covariance matrix.

## Singular Value Decomposition (SVD)
Every non-quadratic matrix can be decomposed as $$X = U \, S \, V^T \, ,$$ where $U$ is $n \times m$ matrix with orthonormal columns ($U^T \, U = I$), $V$ is $m \times m$ orthonormal matrix ($V^T \, V = I$) and $S$ a diagonal $m\times m$ matrix of singular values. From such decomposition one can write
\begin{align}
X \, X^T &= U \, S^2 \, U^T  \, , \\
X^T X &= V \, S^2 \, V^T \, .
\end{align}
Both $X \, X^T$ and $X^T X$ have the same eigenvalues, with larger of the two having the rest equal to $0$.

## Case #1: $n < m$
Covariance of the data can be written as $$ C = \frac{1}{m-1} X \, X^T = U \, \frac{S^2}{m-1} \, U^T \, .$$ Therefore, by solving an eigenvalue problem for $C$, we can find $U$. By picking $N$ eigenvectors in the directions of the largest eigenvalues, we construct $\widetilde{U}$ used for PCA.

For some matrix $X_0$ of size $n \times m_0$, PCA is simply $$ Y_0 = \widetilde{U}^T X_0$$.

```python
N_dim, N_samples = (16, 10**5)
pca = spax.PCA_m(5, devices = jax.devices("gpu"))
data = np.random.normal(0, 1, size = (N_dim, N_samples)) * np.sqrt(np.arange(1, N_dim + 1))[:, np.newaxis]
tic = datetime.now()
pca.fit(data, batch_size = N_dim // 2, centering_data = "GPU") # N_dim % (N_devices * batch_size) == 0
print("DURATION:", datetime.now() - tic)
sampled_data = pca.sample(N_samples, batch_size = N_dim // 2)
print(np.std(pca.transform(data, batch_size = N_dim // 2), axis = 1)**2) # should be [16, 15, 14, 13, 12]
print(np.std(pca.transform(sampled_data, batch_size = N_dim // 2), axis = 1)**2) #should be the same
print(pca.λ ** 2) # should be the same
print(np.round(pca.U.T, 1)) # should be a +-unit matrix on last 5 dimensions
```

    DURATION: 0:00:03.665672
    [15.86349 15.01729 14.025   12.97456 11.96139]
    [15.88904 15.04109 14.00261 12.92626 12.06539]
    [15.86442 15.01822 14.02536 12.97476 11.96212]
    [[ 0.  -0.  -0.  -0.   0.  -0.   0.  -0.   0.  -0.   0.   0.   0.   0.  -0.1  1. ]
     [-0.  -0.  -0.   0.  -0.  -0.   0.  -0.   0.   0.  -0.  -0.  -0.  -0.1  1.   0.1]
     [-0.  -0.  -0.  -0.  -0.   0.  -0.  -0.   0.  -0.  -0.   0.  -0.1  1.   0.1 -0. ]
     [ 0.  -0.  -0.  -0.   0.  -0.   0.   0.   0.   0.  -0.  -0.   1.   0.1  0.  -0. ]
     [ 0.   0.   0.   0.   0.  -0.  -0.  -0.  -0.  -0.   0.   1.   0.  -0.   0.   0. ]]

```python
#testing the result with scikit-learn
pca_sk = PCA(n_components = 5)
tic = datetime.now()
pca_sk.fit(data.T)
print("DURATION:", datetime.now() - tic)
print(np.std(pca_sk.transform(data.T), axis = 0)**2)
print(np.std(pca_sk.transform(sampled_data.T), axis = 0)**2)
print(pca_sk.singular_values_**2 / (N_samples - 1))
print(np.round(pca_sk.components_, 1))
```

    DURATION: 0:00:00.258082
    [15.86399 15.01809 14.02532 12.97499 11.96181]
    [15.88819 15.03968 14.00197 12.92547 12.06467]
    [15.86415 15.01824 14.02546 12.97512 11.96193]
    [[-0.  -0.   0.   0.  -0.   0.   0.   0.  -0.   0.  -0.  -0.  -0.  -0.   0.1 -1. ]
     [-0.   0.  -0.   0.  -0.   0.  -0.  -0.  -0.  -0.   0.   0.   0.   0.1 -1.  -0.1]
     [-0.  -0.   0.  -0.  -0.   0.  -0.  -0.   0.  -0.  -0.   0.  -0.1  1.   0.1 -0. ]
     [-0.  -0.   0.  -0.  -0.   0.   0.  -0.  -0.  -0.   0.   0.  -1.  -0.1 -0.   0. ]
     [ 0.  -0.  -0.   0.  -0.  -0.   0.  -0.  -0.  -0.   0.   1.   0.  -0.   0.   0. ]]

## Case #2 $n \ge m$
In this case, it is better to write: $$D = \frac{1}{n} X^T \, X = V \, \frac{S^2}{n} \, V^T \, .$$
Solving eigenvector problem for $D$ gives us $V$ and $S$. Then, rotation matrix can be computed as $$ U = X \, V \, S^{-1} \, .$$ The rest is the same as in previous case.

```python
N_dim, N_samples = (10**5, 16)
pca = spax.PCA_m(5, devices = jax.devices("gpu"))
data = np.random.normal(0, 1, size = (N_dim, N_samples)) * np.sqrt(np.arange(1, N_dim + 1))[:, np.newaxis]
tic = datetime.now()
pca.fit(data, batch_size = N_dim // 2, centering_data = "GPU") # N_dim % (N_devices * batch_size) == 0
print("DURATION:", datetime.now() - tic)
sampled_data = pca.sample(10**4, batch_size = N_dim // 2)
print(np.std(pca.transform(data, batch_size = N_dim // 2), axis = 1)**2)
print(np.std(pca.transform(sampled_data, batch_size = N_dim // 2), axis = 1)**2)
print(pca.λ ** 2)
print(pca.U.T)
```

    DURATION: 0:00:01.705780
    [3.20480e+08 3.18571e+08 3.17954e+08 3.16769e+08 3.16200e+08]
    [3.41459e+08 3.37940e+08 3.43414e+08 3.37691e+08 3.44295e+08]
    [3.41847e+08 3.39811e+08 3.39152e+08 3.37887e+08 3.37281e+08]
    [[ 6.33442e-06  8.16398e-06 -3.66614e-06 ...  3.28223e-03 -8.05994e-03  2.22728e-03]
     [ 4.32146e-06  2.09783e-05 -5.39127e-06 ...  4.88083e-03 -3.92408e-03  1.12816e-03]
     [-4.10747e-05  4.59198e-06  3.08347e-05 ...  4.53058e-03 -6.94469e-03 -3.38505e-03]
     [ 4.58250e-07  1.24855e-05  1.09394e-05 ...  5.18696e-03  9.30697e-04 -6.11469e-04]
     [-1.19243e-06 -2.17123e-05 -4.26089e-05 ... -3.43697e-03  1.93229e-03 -2.56672e-03]]

```python
pca_sk = PCA(n_components = 5)
tic = datetime.now()
pca_sk.fit(data.T)
print("DURATION:", datetime.now() - tic)
print(np.std(pca_sk.transform(data.T), axis = 0)**2)
print(np.std(pca_sk.transform(sampled_data.T), axis = 0)**2)
print(pca_sk.singular_values_**2 / (N_samples - 1))
print(pca_sk.components_)
```

    DURATION: 0:00:00.166816
    [3.20481e+08 3.18571e+08 3.17954e+08 3.16769e+08 3.16200e+08]
    [3.41459e+08 3.37940e+08 3.43414e+08 3.37690e+08 3.44296e+08]
    [3.41846e+08 3.39809e+08 3.39151e+08 3.37887e+08 3.37281e+08]
    [[ 6.33356e-06  8.16345e-06 -3.66521e-06 ...  3.28216e-03 -8.06003e-03  2.22716e-03]
     [-4.31963e-06 -2.09781e-05  5.39206e-06 ... -4.88114e-03  3.92440e-03 -1.12789e-03]
     [ 4.10754e-05 -4.59016e-06 -3.08327e-05 ... -4.53029e-03  6.94427e-03  3.38534e-03]
     [ 4.60913e-07  1.24849e-05  1.09398e-05 ...  5.18659e-03  9.31135e-04 -6.11179e-04]
     [ 1.18826e-06  2.17128e-05  4.26083e-05 ...  3.43740e-03 -1.93294e-03  2.56633e-03]]

## Test save + load

```python
pca.save("test.hdf5")
pca_new = spax.PCA_m(5, devices = jax.devices("gpu"))
pca_new.load("test.hdf5")

x0 = np.arange(100000)[:, np.newaxis]
print(pca.transform(x0, batch_size = N_dim // 2).flatten())
print(pca_new.transform(x0, batch_size = N_dim // 2).flatten())
print(pca_sk.transform(x0.T).flatten())
```

    [-71815.555   29139.97     3841.4258 -87116.92    23392.324 ]
    [-71815.555   29139.967    3841.4229 -87116.92    23392.318 ]
    [-71812.21997 -29130.60387  -3838.60998 -87106.1801  -23407.71851]


## Quantum Mechanics: First-order energy perturbation to harmonic oscillator

In *Quantum Chemistry, 7th Ed* by Levine, an example on page 235 calculates the first-order perturbation correction to an unperturbed harmonic oscillator. It is a simple example, which I will reproduce here with SymPy. Usually, I wouldn't post something so trivial, but there was a trick using SymPy to get the integration to work. This trick was defining assumptions about variables ahead of time, and I demonstrate that here.

An example perturbed Hamiltonian for an osicalltor incorporating anharmonic terms:

$$ \hat{H}^{'} = \hat{H} - \hat{H}^{0} = cx^3 + dx^4 $$

The ground state unperturbed wavefunction of the quantum harmonic oscillator:

$$ \psi_{0}^{(0)} = (\alpha / \pi)^{1/4} e^{-\alpha x^2 / 2} $$

A little shortcut in notation to simplify equation writing:

$$ \alpha = \frac{4 \pi^2 \nu m}{h} $$

```python
from sympy import symbols, oo, integrate, exp, sqrt, pi
```

Here was the trick to making the integration work properly. When creating the symbols, I defined all as real and most as positive.

```python
alpha, x, m, nu, h = symbols('alpha x m nu h', real=True, positive=True)
d = symbols('d', real=True)
```

```python
E_0_1 = 2*d*sqrt(alpha/pi) * integrate(exp(-alpha*x**2)*x**4, (x, 0, oo))
E_0_1
```

$\displaystyle \frac{3 d}{4 \alpha^{2}}$

```python
E_0_1 = E_0_1.subs({alpha: (4*pi**2*nu*m/h)})
E_0_1
```

$\displaystyle \frac{3 d h^{2}}{64 \pi^{4} m^{2} \nu^{2}}$

```python
E_0_0 = 0.5 * h * nu
E_0_0
```

$\displaystyle 0.5 h \nu$

```python

```


# Statistics

## 1.1 Sales for the past week was the following amounts: [3505, 2400, 3027, 2798, 3700, 3250, 2689]. Without using library functions, what is the mean, variance, and standard deviation of of sales from last week? (for extra bonus points, write your own function that can calculate these two values for any sized list)

```
import numpy as np

sales = [3505, 2400, 3027, 2798, 3700, 3250, 2689]
mean = sum(sales) / len(sales)

def var(n):
    l = []
    m = sum(n) / len(n)
    for i in n:
        l.append((i - m) ** 2)
    return sum(l)/len(l)

def stdev(n):
    l = []
    m = sum(n) / len(n)
    for i in n:
        l.append((i - m) ** 2)
    return (sum(l) / len(l)) ** .5

print("The mean is:",mean,"\nThe variance is:",var(sales),"\nthe standard deviation is:",stdev(sales))

```

    The mean is: 3052.714285714286
    The variance is: 183761.06122448976
    the standard deviation is: 428.67360686714756

## 1.2 Find the covariance between last week's sales numbers and the number of customers that entered the store last week: [127, 80, 105, 92, 120, 115, 93] (you may use librray functions for calculating the covariance since we didn't specifically talk about its formula)

```
#using numpy for covariance output

customers = [127, 80, 105, 92, 120, 115, 93]

np.cov(sales,customers)
```

    array([[214387.9047619 ,   7604.35714286],
           [  7604.35714286,    290.95238095]])

```
#creating dataframe
data = {"sales": [3505, 2400, 3027, 2798, 3700, 3250, 2689], "entered": [127, 80, 105, 92, 120, 115, 93]}

df = pd.DataFrame(data)
df
```

      sales
      entered

      0
      3505
      127

      1
      2400
      80

      2
      3027
      105

      3
      2798
      92

      4
      3700
      120

      5
      3250
      115

      6
      2689
      93

###Covariance: need to explain to oneself

```
#question: what only two values per intersection? what does this interaction actually saying?

#calculating covariance via pandas

cov = df.cov()
cov
```

      sales
      entered

      sales
      214387.904762
      7604.357143

      entered
      7604.357143
      290.952381

## 1.3 Find the standard deviation of customers who entered the store last week. Then, use the standard deviations of both sales and customers to standardize the covariance to find the correlation coefficient that summarizes the relationship between sales and customers. (You may use library functions to check your work.)

```
import statistics

#not getting the correct output from running this manually and also what is wrong with my standard deviation

def stdev(n):
    l = []
    m = sum(n) / (len(n) - 1)
    for i in n:
        l.append((i - m) ** 2)
    return (sum(l) / len(l)) ** .5

print("custy stdev:",stdev(customers),"sales stdev:",stdev(sales))

prod1 = stdev(customers) * stdev(sales)

prod2 = statistics.stdev(customers) * statistics.stdev(sales)

display(cov.div(prod1),cov.div(prod2))

```

    custy stdev: 23.518989288293344 sales stdev: 665.2999055205962

      sales
      entered

      sales
      13.701375
      0.485989

      entered
      0.485989
      0.018595

      sales
      entered

      sales
      27.144959
      0.962834

      entered
      0.962834
      0.036839

```
# 214387.904762 / prod

prod
```

    7897.89030930846

```
#The correct correlation coefficients

cov.corr()
```

      sales
      entered

      sales
      1.0
      1.0

      entered
      1.0
      1.0

## 1.4 Use pandas to import a cleaned version of the titanic dataset from the following link: [Titanic Dataset](https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv)

## Calculate the variance-covariance matrix and correlation matrix for the titanic dataset's numeric columns. (you can encode some of the categorical variables and include them as a stretch goal if you finish early)

```
dft = pd.read_csv("https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv")
```

```
dft = dft.fillna(0)
dft.head()
```

      Unnamed: 0
      pclass
      survived
      name
      sex
      age
      sibsp
      parch
      ticket
      fare
      cabin
      embarked
      boat
      body
      home.dest
      has_cabin_number

      0
      1
      1.0
      1.0
      Allen, Miss. Elisabeth Walton
      female
      29.0000
      0.0
      0.0
      24160
      211.3375
      B5
      S
      2
      0.0
      St Louis, MO
      1

      1
      2
      1.0
      1.0
      Allison, Master. Hudson Trevor
      male
      0.9167
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      11
      0.0
      Montreal, PQ / Chesterville, ON
      1

      2
      3
      1.0
      0.0
      Allison, Miss. Helen Loraine
      female
      2.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      0
      0.0
      Montreal, PQ / Chesterville, ON
      1

      3
      4
      1.0
      0.0
      Allison, Mr. Hudson Joshua Creighton
      male
      30.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      0
      135.0
      Montreal, PQ / Chesterville, ON
      1

      4
      5
      1.0
      0.0
      Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
      female
      25.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      0
      0.0
      Montreal, PQ / Chesterville, ON
      1

```
dft2 = dft.drop(['name', 'sex',"age","cabin","embarked","boat","home.dest","has_cabin_number"],axis=1)
dft2.head()
```

      Unnamed: 0
      pclass
      survived
      sibsp
      parch
      ticket
      fare
      body

      0
      1
      1.0
      1.0
      0.0
      0.0
      24160
      211.3375
      0.0

      1
      2
      1.0
      1.0
      1.0
      2.0
      113781
      151.5500
      0.0

      2
      3
      1.0
      0.0
      1.0
      2.0
      113781
      151.5500
      0.0

      3
      4
      1.0
      0.0
      1.0
      2.0
      113781
      151.5500
      135.0

      4
      5
      1.0
      0.0
      1.0
      2.0
      113781
      151.5500
      0.0

```
dft2.cov()
```

      Unnamed: 0
      pclass
      survived
      sibsp
      parch
      fare
      body

      Unnamed: 0
      143117.500000
      282.992361
      -54.116883
      25.559587
      0.979374
      -9427.515484
      -613.393430

      pclass
      282.992361
      0.705453
      -0.126482
      0.053924
      0.013952
      -24.150423
      -2.164033

      survived
      -54.116883
      -0.126482
      0.236181
      -0.013932
      0.034861
      6.156049
      -5.673581

      sibsp
      25.559587
      0.053924
      -0.013932
      1.084413
      0.336722
      8.653922
      -3.048352

      parch
      0.979374
      0.013952
      0.034861
      0.336722
      0.748735
      9.932434
      -1.153653

      fare
      -9427.515484
      -24.150423
      6.156049
      8.653922
      9.932434
      2676.557808
      -40.679167

      body
      -613.393430
      -2.164033
      -5.673581
      -3.048352
      -1.153653
      -40.679167
      3044.601879

```
dft2.corr()
```

      Unnamed: 0
      pclass
      survived
      sibsp
      parch
      fare
      body

      Unnamed: 0
      1.000000
      0.890624
      -0.294350
      0.064880
      0.002992
      -0.481684
      -0.029385

      pclass
      0.890624
      1.000000
      -0.309865
      0.061652
      0.019197
      -0.555779
      -0.046694

      survived
      -0.294350
      -0.309865
      1.000000
      -0.027529
      0.082901
      0.244845
      -0.211578

      sibsp
      0.064880
      0.061652
      -0.027529
      1.000000
      0.373689
      0.160630
      -0.053052

      parch
      0.002992
      0.019197
      0.082901
      0.373689
      1.000000
      0.221872
      -0.024163

      fare
      -0.481684
      -0.555779
      0.244845
      0.160630
      0.221872
      1.000000
      -0.014250

      body
      -0.029385
      -0.046694
      -0.211578
      -0.053052
      -0.024163
      -0.014250
      1.000000

# Orthogonality

## 2.1 Plot two vectors that are orthogonal to each other. What is a synonym for orthogonal?

```
vector_1 = [0, 1]
vector_2 = [1, 0]

# Plot the Scaled Vectors
plt.arrow(0,0, vector_1[0], vector_1[1],head_width=.05, head_length=0.05, color ='orange')
plt.arrow(0,0, vector_2[0], vector_2[1],head_width=.05, head_length=0.05, color ='green')
plt.xlim(-1,3)
plt.ylim(-1,3)
plt.title("Orthogonal Vectors AKA PERPENDICULAR Vectors")
plt.show()
```

```
#clearly shows these two are orthogonal:

np.dot(vector_1,vector_2)
```

    0

## 2.2 Are the following vectors orthogonal? Why or why not?

\begin{align}
a = \begin{bmatrix} -5 \\ 3 \\ 7 \end{bmatrix}
\qquad
b = \begin{bmatrix} 6 \\ -8 \\ 2 \end{bmatrix}
\end{align}

```
#calculating dot product of the two vectors
a = [-5,3,7]
b = [6,-8,2]

print("vecotrs a and b are not orthogonal because their dot product does not equal zero:",np.dot(a,b))
```

    vecotrs a and b are not orthogonal because their dot product does not equal zero: -40

## 2.3 Compute the following values: What do these quantities have in common?

## What is $||c||^2$?

## What is $c \cdot c$?

## What is $c^{T}c$?

\begin{align}
c = \begin{bmatrix} 2 & -15 & 6 & 20 \end{bmatrix}
\end{align}

```
#NEED HELP WITH THIS

from numpy import linalg as LA

c = [2,-15,6,20]

print("these values are all identical because it is performing the same calculation \n","squared norm of c:", LA.norm(c)**2,"\ndot product of c,c:",np.dot(c,c),"\ndot product of c transposed and c:",np.dot(np.transpose(c),c))

```

    these values are all identical because it is performing the same calculation
     squared norm of c: 665.0
    dot product of c,c: 665
    dot product of c transposed and c: 665

# Unit Vectors

## 3.1 Using Latex, write the following vectors as a linear combination of scalars and unit vectors:

\begin{align}
d = \begin{bmatrix} 7 \\ 12 \end{bmatrix}
\qquad
e = \begin{bmatrix} 2 \\ 11 \\ -8  \end{bmatrix}
\end{align}

###(Zhenya) Answer

\begin{align}
c = 7\hat{i} + 12\hat{j}
\end{align}

\begin{align}
c = 2\hat{i} + 11\hat{j} + -8\hat{k}
\end{align}

## 3.2 Turn vector $f$ into a unit vector:

\begin{align}
f = \begin{bmatrix} 4 & 12 & 11 & 9 & 2 \end{bmatrix}
\end{align}

```
#IS THIS CORRECT?

#Can turn any vector inot a unit vector by dividing by the norm

f = [4,12,11,9,2]

fn = np.linalg.norm(f)

f / fn
```

    array([0.20908335, 0.62725005, 0.57497921, 0.47043754, 0.10454167])

# Linear Independence / Dependence

## 4.1 Plot two vectors that are linearly dependent and two vectors that are linearly independent (bonus points if done in $\mathbb{R}^3$).

```
#how do you plot an R3 version of this?!

# Plot Linearly Dependent Vectors

# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
v = [1,0]

# Scaled Vectors
v2 = np.multiply(3, v)
v3 = np.multiply(-1,v)

# Get Vals for L
axes = plt.gca()
x_vals = np.array(axes.get_xlim())
y_vals = 0*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0,0, v2[0], v2[1], linewidth=3, head_width=.05, head_length=0.05, color ='yellow')
plt.arrow(0,0, v[0], v[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.arrow(0,0, v3[0], v3[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("Linearly Dependent Vectors")
plt.show()
```

```
# Plot Linearly Independent Vectors

# Axis Bounds
plt.xlim(-2,3.5)
plt.ylim(-1,3)

# Original Vector
a = [-1.5,.5]
b = [3, 1]

# Plot Vectors
plt.arrow(0,0, a[0], a[1], linewidth=3, head_width=.05, head_length=0.05, color ='blue')
plt.arrow(0,0, b[0], b[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("Linearly Independent Vectors")
plt.show()
```

# Span

## 5.1 What is the span of the following vectors?

\begin{align}
g = \begin{bmatrix} 1 & 2 \end{bmatrix}
\qquad
h = \begin{bmatrix} 4 & 8 \end{bmatrix}
\end{align}

```
#These are a single R because they scale
```

## 5.2 What is the span of $\{l, m, n\}$?

\begin{align}
l = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}
\qquad
m = \begin{bmatrix} -1 & 0 & 7 \end{bmatrix}
\qquad
n = \begin{bmatrix} 4 & 8  & 2\end{bmatrix}
\end{align}

```
#the span here is R^3 because neither vector is in the same dimension
```

# Basis

## 6.1 Graph two vectors that form a basis for $\mathbb{R}^2$

```
# Plot Linearly Independent Vectors

# Axis Bounds
plt.xlim(-2,3.5)
plt.ylim(-1,3)

# Original Vector
a = [-1.5,.5]
b = [3, 1]

# Plot Vectors
plt.arrow(0,0, a[0], a[1], linewidth=3, head_width=.05, head_length=0.05, color ='blue')
plt.arrow(0,0, b[0], b[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("Linearly Independent Vectors")
plt.show()
```

## 6.2 What does it mean to form a basis?

# Rank

## 7.1 What is the Rank of P?

\begin{align}
P = \begin{bmatrix}
1 & 2 & 3 \\
 -1 & 0 & 7 \\
4 & 8  & 2
\end{bmatrix}
\end{align}

## 7.2 What does the rank of a matrix tell us?

# Linear Projections

## 8.1 Line $L$ is formed by all of the vectors that can be created by scaling vector $v$
\begin{align}
v = \begin{bmatrix} 1 & 3 \end{bmatrix}
\end{align}

\begin{align}
w = \begin{bmatrix} -1 & 2 \end{bmatrix}
\end{align}

## find $proj_{L}(w)$

## graph your projected vector to check your work (make sure your axis are square/even)

```

```

# Stretch Goal

## For vectors that begin at the origin, the coordinates of where the vector ends can be interpreted as regular data points. (See 3Blue1Brown videos about Spans, Basis, etc.)

## Write a function that can calculate the linear projection of each point (x,y) (vector) onto the line y=x. run the function and plot the original points in blue and the new projected points on the line y=x in red.

## For extra points plot the orthogonal vectors as a dashed line from the original blue points to the projected red points.

```
import pandas as pd
import matplotlib.pyplot as plt

# Creating a dataframe for you to work with -Feel free to not use the dataframe if you don't want to.
x_values = [1, 4, 7, 3, 9, 4, 5 ]
y_values = [4, 2, 5, 0, 8, 2, 8]

data = {"x": x_values, "y": y_values}

df = pd.DataFrame(data)

df.head()

plt.scatter(df.x, df.y)
plt.show()
```

```

```


# Projectile motion (Part 1)
Let's apply our ODE solvers to some problems involving balls and projectiles.

The `integrators.py` file from section 11 is used here (and named `ode.py`).

```python
import numpy as np
import ode
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

## Projectile with linear air-resistance
Linear drag force

$$
\mathbf{F}_1 = -b_1 \mathbf{v}
$$

Equations of motion with force due to gravity $\mathbf{g} = -g \hat{\mathbf{e}}_y$

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= - g \hat{\mathbf{e}}_y -\frac{b_1}{m} \mathbf{v}
\end{align}

Bring into standard ODE form for

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$

as

$$
\mathbf{y} = \begin{pmatrix}
x\\
y\\
v_x\\
v_y
\end{pmatrix}, \quad
\mathbf{f} = \begin{pmatrix}
v_x\\
v_y\\
-\frac{b_1}{m} v_x\\
-g -\frac{b_1}{m} v_y
\end{pmatrix}
$$

(Based on Wang 2016, Ch 3.3.1)

```python
def simulate(v0, h=0.01, b1=0.2, g=9.81, m=0.5):

    def f(t, y):
        # y = [x, y, vx, vy]
        return np.array([y[2], y[3], -b1/m * y[2], -g - b1/m * y[3]])

    vx, vy = v0
    t = 0
    positions = []
    y = np.array([0, 0, vx, vy], dtype=np.float64)

    while y[1] >= 0:
        positions.append([t, y[0], y[1]])  # record t, x and y
        y[:] = ode.rk4(y, f, t, h)
        t += h

    return np.array(positions)

def initial_v(v, theta):
    x = np.deg2rad(theta)
    return v * np.array([np.cos(x), np.sin(x)])
```

```python
r = simulate(initial_v(200, 30), h=0.01, b1=1)
```

```python
plt.plot(r[:, 1], r[:, 2])
```

```python
for angle in (5, 7.5, 10, 20, 30, 45):
    r = simulate(initial_v(200, angle), h=0.01, b1=1)
    plt.plot(r[:, 1], r[:, 2], label=r"$\theta = {}^\circ$".format(angle))
plt.legend(loc="best")
plt.xlabel("x (m)")
plt.ylabel("y (m)")
```

## Simple Baseball physics

- quadratic air resistance (with velocity-dependent drag coefficient)
- Magnus force due to spin

### Quadratic air resistance
Occurs at high Reynolds numbers, i.e., turbulent flow. Only approximate:

$$
\mathbf{F}_2 = -b_2 v \mathbf{v}
$$

### Magnus effect

**Magnus effect**: airflow is changed around a spinning object. The Magnus force is

$$
\mathbf{F}_M = \alpha \boldsymbol{\omega} \times \mathbf{v}
$$

where $\boldsymbol{\omega}$ is the ball's angular velocity in rad/s (e.g., 200/s for a baseball).

For a sphere the proportionality constant $\alpha$ can be written

$$
\mathbf{F}_M = \frac{1}{2} C_L \rho A \frac{v}{\omega} \boldsymbol{\omega} \times \mathbf{v}
$$

where $C_L$ is the lift coefficient, $\rho$ the air density, $A$ the ball's cross section. (Advantage of defining $C_L$ this way: when spin and velocity are perpendicular, the Magnus force is simply $F_M = \frac{1}{2} C_L \rho A v^2$.)

$C_L$ is mainly a function of the *spin parameter*

$$
S = \frac{r\omega}{v}
$$

with the radius $r$ of the ball. In general we write

$$
\mathbf{F}_M = \frac{1}{2} C_L  \frac{\rho A r}{S} \boldsymbol{\omega} \times \mathbf{v}
$$

For a baseball, experimental data show approximately a power law dependence of $C_L$ on $S$

$$
C_L = 0.62 \times S^{0.7}
$$

All together:

\begin{align}
\mathbf{F}_M &= \alpha\ \boldsymbol{\omega} \times \mathbf{v}\\
v &= \sqrt{\mathbf{v}}\\
S &= \frac{r\omega}{v}\\
C_L &= 0.62 \times S^{0.7}\\
\alpha &= \frac{1}{2} C_L  \frac{\rho A r}{S}
\end{align}

### Equations of motion

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= -g \hat{\mathbf{e}}_y \mathbf{v} -\frac{b_2}{m} v \mathbf{v} + \alpha\ \boldsymbol{\omega} \times \mathbf{v}\\
\end{align}

(quadratic drag $-\frac{b_2}{m} v \mathbf{v}$ included.)

### Baseball simulation

Implement the full baseball equations of motions:
- gravity $a_\text{gravity}$
- quadratic drag $a_\text{drag}$
- Magnus effect $a_\text{Magnus}$

For the cross product you can look at [numpy.cross()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cross.html).

```python
def C_L(S):
    return 0.62 * S**0.7

def simulate_baseball(v0, omega=200.*np.array([0,1,1]), r0=np.array([0, 2.]),
                      h=0.01, b2=0.0013, g=9.81, rho = 1.225,
                      r=0.07468/2, m=0.14883, R_homeplate=18.4):

    # make sure that omega is a numpy array
    omega = np.asarray(omega)

    # all SI units (kg, m)
    # air density rho in kg/m^3

    domega = np.linalg.norm(omega)
    A = np.pi*r**2
    rhoArm = rho * A * r / m

    # internally, use 3d coordinates [x,y,z];
    # y = [x, y, z, vx, vy, vz]

    a_gravity = np.array([0, -g, 0])

    def f(t, y):
        # y = [x, y, z, vx, vy, vz]
        v = y[3:]
        dv = np.linalg.norm(v)
        # COMPLETE
        # 1. acceleration due to drag
        # 2. acceleration due to Magnus effect
        # 3. acceleration due to gravity (a_gravity)

        # need to return array f of length 6!
        raise NotImplementedError

    x0, y0 = r0
    vx, vy = v0
    t = 0
    positions = []
    # initialize 3D!
    y = np.array([x0, y0, 0, vx, vy, 0], dtype=np.float64)

    # IMPLEMENT integration loop
    # - use ode.rk4()
    # - stop when x >= R_homeplate or y < 0.2 (i.e. cannot be caught)

    return np.array(positions)

```

#### Simulate throws

Simulate baseball throw for initial velocity $\mathbf{v} = (30\,\text{m/s}, 0)$.

Plot x vs y and x vs z (to see curving).

Try out different spins; a good value is $\boldsymbol{\omega} = 200\,\text{rad/s} \times (0, 1, 1)$.

Simulate the baseball throw with
- almost no spin: $\omega = 0.001 \times (0, 0, 1)$  (our code does not handle $\omega = 0$ gracefully...)
- full upward spin: $\omega = 200 \times (0, 0, 1)$
- sideways spin: $\omega = 200 \times (0, 1, 1)$

```python
r = simulate_baseball([30, 0], omega=0.001*np.array([0,0,1]))
rz = simulate_baseball([30, 0], omega=200.*np.array([0,0,1]))
rzy = simulate_baseball([30, 0], omega=200.*np.array([0,1,1]))
```

#### Plotting

Plot the three scenarios in 2D planes: x-y (side view) and x-z (top view).

#### 3D plot
Use simple `matplotlib` 3D plot. (BONUS: Make it work with vpython)

If we use the [`notebook` backend for matplotlib](http://ipython.readthedocs.io/en/stable/interactive/plotting.html) then we will be able to interactively rotate our [matplotlib 3D graphics](http://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html). (Note: If this does not seem to work, disable adblockers and allow javascript on the page.)

```python
%matplotlib notebook
```

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(r[:,1], r[:,3], r[:,2], 'o', label="no spin")
# ...

# hand of the catcher, 0.2m above homeplate
ax.plot([18.4, 18.4], [0, 0], [0, 0.2], color="black", lw=6)

ax.set_xlabel("$x$ (m)")
ax.set_ylabel("$z$ (m)")
ax.set_zlabel("$y$ (m)")
ax.legend(loc="upper left", numpoints=1)
ax.figure.tight_layout()
```

```python

```

## Reynolds number

$$
\text{Re} = \frac{\rho v L}{\mu}\\
\text{Re} > 2300\quad\text{flow turbulent}
$$

* density $\rho$: air  1.275 kg/m^3 (kilograms per cubic meter)
* fluid viscosity $\mu$: air 1.845×10^-5 Pa s (pascal seconds) (at 25 °C)

(from Wolfram Alpha)

```python
rho_air = 1.275    # kg/m^3
mu_air = 1.845e-5  # Pa s
```

```python
L = 0.05  # m
v = 200   # m/s
```

```python
def ReynoldsNumber(v, L, rho=rho_air, mu=mu_air):
    return rho*v*L/mu
```

```python
ReynoldsNumber(v, L)
```

    691056.9105691056

This means that we really should be using quadratic air resistance for the projectile.

```python

```


# Papoulis

    TODO

```python
import sympy
import sympy.functions.elementary.exponential as symExp
from sympy.plotting import plot
import numpy as np
import matplotlib.pyplot as plt
from visuals import *
```

```python
c, t, U = sympy.symbols('c, t, U')
```

```python
f = c*symExp.exp(-c*t)
f
```

    c*exp(-c*t)

```python
int_f = sympy.integrate(f, (t, 0, t))
int_f
sympy.oo
```

    oo

```python
q = sympy.integrate(f, t)
q
```

    -exp(-c*t)

sympy.limit(q, t, sympy.oo)

$\infty$

```python
temp = int_f.subs({c:1, U:1})
temp
```

    1 - exp(-t)

```python
p = plot(temp, (t, -2, 5), show=False)
p.show()
```

```python
from sympy.utilities.lambdify import lambdify
func = lambdify(t, temp, 'numpy') # returns a numpy-ready function1
```

```python

interval = np.linspace(-1, 5, num=1000)
values = func(interval)
```

def pdfInfo(interval, values, rng, xlim=None):
    fig, ax = plt.subplots()
    ax.plot(interval, values, 'k')

    if not np.isscalar(rng) and len(rng)==1:
        rng = rng[0]

    if not np.isscalar(rng):
        x_1 = np.argmax(interval>rng[0])
        x_2 = np.argmax(interval>rng[1])
        ax.fill_between(np.linspace(rng[0], rng[1], num=len(values[x_1:x_2])), values[x_1:x_2], color='k')
    else:
        x = np.argmax(interval>rng)
        x_0 = interval[0]
        ax.fill_between(np.linspace(x_0, rng, num=len(values[0:x])), values[0:x], color='k')

    ax.axhline(0, c='k', alpha=.5)
    ax.axvline(0, c='k', alpha=.5)

    ax.yaxis.grid(b=True, which='minor', color='k', linestyle='--', alpha=.2, zorder=0)
    ax.yaxis.grid(b=True, which='major', color='k', linestyle='--', alpha=.5, zorder=0)

    ax.set_facecolor('xkcd:white')
    ax.spines["top"].set_visible(False)
    ax.spines["right"].set_visible(False)
    ax.spines["left"].set_visible(False)
    ax.spines["bottom"].set_visible(False)

    if xlim != None:
         ax.set_xlim(xlim)

```python
pdfInfo(interval, values, [2, 3], xlim=[-2, 7])
```

```python

```

```python

```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sy
from openmm import unit
```

# The Lennard-Jones potential

The Lennard-Jones 12-6 potential between two particles is defined by the following equation, where $x$ is the distance between the particles, and $\sigma$ and $\epsilon$ are two parameters of the potential:

$$
V(x) = 4 \epsilon \left[ \left( \frac{\sigma}{x} \right)^{12} - \left( \frac{\sigma}{x} \right)^6 \right]
$$ (LJ_potential)

Lets see the shape of this function:

```python
def LJ (x, sigma, epsilon):

    t = sigma/x
    t6 = t**6
    t12 = t6**2

    return 4.0*epsilon*(t12-t6)
```

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

xlim_figure = [0.01, 6.0]
ylim_figure = [-2.0, 10.0]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstrom
plt.plot(x, LJ(x, sigma, epsilon))
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

The way the LJ potential is built, the $\sigma$ and $\epsilon$ parameters have a straightforward interpretation. The cut with $y=0$ is located in $x=\sigma$:

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

xlim_figure = [0.01, 6.0]
ylim_figure = [-2.0, 10.0]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstrom
plt.plot(x, LJ(x, sigma, epsilon))
plt.hlines(0, xlim_figure[0], xlim_figure[1], linestyles='dotted', color='gray')
plt.vlines(sigma._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='red')
plt.text(sigma._value+0.02*xlim_figure[1], 0.7*ylim_figure[1], '$\sigma$', fontsize=14)
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

And $\epsilon$ is the depth of the minimum measured from $y=0$:

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

xlim_figure = [0.01, 6.0]
ylim_figure = [-2.0, 10.0]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstrom
plt.plot(x, LJ(x, sigma, epsilon))
plt.hlines(0, xlim_figure[0], xlim_figure[1], linestyles='dotted', color='gray')
plt.hlines(-epsilon._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='red')
plt.annotate(text='', xy=(1.0,0.0), xytext=(1.0,-epsilon._value), arrowprops=dict(arrowstyle='<->'))
plt.text(1.0+0.02*xlim_figure[1], -0.7*epsilon._value, '$\epsilon$', fontsize=14)
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

Notice that the LJ potential has physical meaning when $\epsilon>0$ and $\sigma>0$ only. Actually, the potential vanishes whether $\epsilon=0$ or $\sigma=0$.

## The Lennard Jones minimum and the size of the particles

The LJ potential has a single minimum located in $x_{min}$. Lets equal to $0$ the first derivative of the potential to find the value of $x_{min}$:

```python
x, sigma, epsilon = sy.symbols('x sigma epsilon', real=True, positive=True)
V = 4.0*epsilon*((sigma/x)**12-(sigma/x)**6)
gradV = sy.diff(V,x)
roots=sy.solve(gradV, x)
x_min = roots[0]
```

```python
x_min
```

$\displaystyle 1.12246204830937 \sigma$

The minimum is then located in:

$$
x_{min} = 2^{1/6} \sigma
$$ (LJ_min)

where the potential takes the value:

$$
V(x_{min}) = -\epsilon
$$ (LJ_min_potential)

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

x_min = 2**(1/6)*sigma
y_min = -epsilon

xlim_figure = [x_min._value-0.4, x_min._value+0.4]
ylim_figure = [y_min._value-0.1, y_min._value+0.5]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstroms
plt.plot(x, LJ(x, sigma, epsilon))
plt.hlines(y_min._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='gray')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

This way two particles in the equilibrium position will be placed at a $2^{1/6} \sigma$ distance. The potential is thereby modeling two "soft spheres" atracting each other very lightly. Their radii, given that both particles are equal, are equal to $r$:

$$
r = \frac{1}{2} x_{min} = 2^{-5/6} \sigma
$$ (LJ_radius)

And we say these spheres are "soft" because their volume is not limited by a hard-wall potential, they can penetrate each other suffering a not infinite repulsive force.

## Time period of the small harmonic oscillations around the minimum

If we want to perform a molecular simulation of this two particles we should wonder how big the integrator timestep must be. To answer this question we can study the harmonic approximation around the minimum. Lets calculate the time period, $\tau$, of a small harmonic oscillation around the minimum:

```python
x, sigma, epsilon = sy.symbols('x sigma epsilon', real=True, positive=True)
V = 4.0*epsilon*((sigma/x)**12-(sigma/x)**6)
gradV = sy.diff(V,x)
grad2V = sy.diff(V,x,x)

x_min = sy.solve(gradV,x)[0]
k_harm = grad2V.subs(x, x_min)
```

```python
k_harm
```

$\displaystyle \frac{57.1464378708551 \epsilon}{\sigma^{2}}$

The harmonic constant of the second degree Taylor polynomial of the LJ potential at $x=x_{min}$ is then:

$$
k_{harm} = 36·2^{2/3} \frac{\epsilon}{\sigma^2}
$$ (k_harm)

The oscillation period of a particle with $m$ mass in an harmonic potential defined by $\frac{1}{2} k x²$ is:

$$
\tau = 2 \pi \sqrt{ \frac{m}{k}}
$$ (tau)

As such, the period of the small harmonic oscillations around the LJ minimum of particle with $m$ mass is:

$$
\tau = 2 \pi \sqrt{ \frac{m}{k_{harm}}} = \frac{\pi}{3·2^{1/3}} \sqrt{\frac{m\sigma^2}{\epsilon}}
$$ (tau_2)

With the mass and parameters taking values of amus, angstroms and kilocalories per mole, the time period is in the order of:

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

tau = 2*np.pi * np.sqrt(mass/k)

print(tau)
```

    0.5746513694274475 ps

But, is this characteristic time a good threshold for a LJ potential? If the oscillations around the minimum are not small enough, the harmonic potential of the second degree term of the taylor expansion is easily overcome by the sharp left branch of the LJ potential:

```python
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

x_min = 2**(1/6)*sigma
y_min = -epsilon

xlim_figure = [x_min._value-0.2, x_min._value+0.2]
ylim_figure = [y_min._value-0.1, y_min._value+0.6]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstroms
plt.plot(x, LJ(x, sigma, epsilon))
plt.plot(x, 0.5*k*(x-x_min)**2+y_min)
plt.hlines(y_min._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='gray')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

Let's imagine the following situation. Let a particle be in the harmonic potential at temperature of 300K. Will the particle be more constrained in space than in the well of the LJ potential? Will the particle feel the harmonic potential softer or sharper than the LJ? Lets make some numbers to evaluate if the oscillation time period of the harmonic approximation can be a good time threshold for the integration timestep of a molecular dynamics of the LJ potential.

The standard deviation of an harmonic oscillation with the shape $\frac{1}{2}k x^2$ in contact with a stochastic thermal bath can be computed as:

$$
\beta = \frac{1}{k_{\rm B} T}
$$ (beta)

$$
Z_x = \int_{-\infty}^{\infty} {\rm e}^{- \beta \frac{1}{2}k x^2} = \sqrt{\frac{2 \pi}{\beta k}}
$$ (Z_x)

$$
\left< x \right> = \frac{1}{Z_x} \int_{-\infty}^{\infty} x {\rm e}^{-\beta \frac{1}{2}k x^2} = 0
$$ (aver_x)

$$
\left< x^2 \right> = \frac{1}{Z_x} \int_{-\infty}^{\infty} x^{2} {\rm e}^{-\beta \frac{1}{2}k x^2} = \frac{1}{Z_x}  \sqrt{\frac{2 \pi}{\beta³ k^3}} = \frac{1}{\beta k}
$$ (aver_2)

$$
{\rm std} = \left( \left< x^2 \right> -\left< x \right>^2 \right)^{1/2} = \sqrt{ \frac{k_{\rm B}T}{k} }
$$ (std)

This way, in the case of the harmonic potential obtained as the second degree term of the Taylor expansion around the LJ minimum:

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole
temperature = 300 * unit.kelvin
kB = unit.BOLTZMANN_CONSTANT_kB * unit.AVOGADRO_CONSTANT_NA

k = 36 * 2**(2/3) * epsilon/sigma**2
std = np.sqrt(kB*temperature/k)

x_min = 2**(1/6)*sigma
y_min = -epsilon

xlim_figure = [x_min._value-0.4, x_min._value+0.4]
ylim_figure = [y_min._value-0.1, y_min._value+0.6]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstroms
plt.plot(x, LJ(x, sigma, epsilon))
plt.plot(x, 0.5*k*(x-x_min)**2+y_min)
plt.hlines(y_min._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='gray')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.axvspan(x_min._value - std._value, x_min._value + std._value, alpha=0.2, color='red')
plt.annotate(text='', xy=(x_min._value, y_min._value - 0.5*(y_min._value-ylim_figure[0])),
             xytext=(x_min._value-std._value, y_min._value - 0.5*(y_min._value-ylim_figure[0])),
             arrowprops=dict(arrowstyle='<->'))
plt.text(x_min._value-0.6*std._value, y_min._value - 0.4*(y_min._value-ylim_figure[0]), '$std$', fontsize=14)
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

The harmonic potential is too soft as approximation. Its oscillation time used as threshold to choose the integration timestep can yield to numeric problems. Let's try with a stiffer potential, let's double the harmonic constant:

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole
temperature = 300 * unit.kelvin
kB = unit.BOLTZMANN_CONSTANT_kB * unit.AVOGADRO_CONSTANT_NA

k = 36 * 2**(2/3) * epsilon/sigma**2
std = np.sqrt(kB*temperature/k)

x_min = 2**(1/6)*sigma
y_min = -epsilon

xlim_figure = [x_min._value-0.4, x_min._value+0.4]
ylim_figure = [y_min._value-0.1, y_min._value+0.6]

x = np.linspace(xlim_figure[0], xlim_figure[1], 100, True) * unit.angstroms
plt.plot(x, LJ(x, sigma, epsilon))
plt.plot(x, 0.5*k*(x-x_min)**2+y_min)
plt.plot(x, k*(x-x_min)**2+y_min, label='2k_{harm}')
plt.hlines(y_min._value, xlim_figure[0], xlim_figure[1], linestyles='dashed', color='gray')
plt.vlines(x_min._value, ylim_figure[0], ylim_figure[1], linestyles='dashed', color='gray')
plt.axvspan(x_min._value - std._value, x_min._value + std._value, alpha=0.2, color='red')
plt.annotate(text='', xy=(x_min._value, y_min._value - 0.5*(y_min._value-ylim_figure[0])),
             xytext=(x_min._value-std._value, y_min._value - 0.5*(y_min._value-ylim_figure[0])),
             arrowprops=dict(arrowstyle='<->'))
plt.text(x_min._value-0.6*std._value, y_min._value - 0.4*(y_min._value-ylim_figure[0]), '$std$', fontsize=14)
plt.xlim(xlim_figure)
plt.ylim(ylim_figure)
plt.xlabel('x [{}]'.format(x.unit.get_symbol()))
plt.ylabel('V [{}]'.format(epsilon.unit.get_symbol()))
plt.show()
```

Lets take then, as reference, an harmonic potential with constant equal to $2k_{harm}$ could be a better idea. Lets compute then the new time threshold to choose the integration timestep:

$$
\tau' = 2 \pi \sqrt{ \frac{m}{2k_{harm}}} = \frac{\pi}{3·2^{5/6}} \sqrt{\frac{m\sigma^2}{\epsilon}} = \frac{1}{\sqrt{2}} \tau
$$ (tau')

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

tau = 2*np.pi * np.sqrt(mass/(2*k))

print(tau)
```

    0.4063398801402841 ps

It is an accepted rule of thumb that the integration timestep must be as large as $\tau / 10$, being $\tau$ the oscillation time period of the fastest possible vibration mode. So finally, in this case the integration time step should not be longer than:

```python
mass = 50.0 * unit.amu
sigma = 2.0 * unit.angstrom
epsilon = 1.0 * unit.kilocalories_per_mole

k = 36 * 2**(2/3) * epsilon/sigma**2

tau = 2*np.pi * np.sqrt(mass/(2*k))

print(tau/10.0)
```

    0.04063398801402841 ps


# 線形変換処理

## 配列の準備
数値計算ライブラリ NumPy を利用

```
import numpy as np

arrayX = np.matrix([[2,6,2,4],[2,2,2,6],[6,6,4,4],[2,4,4,4]])
display(arrayX)
```

    matrix([[2, 6, 2, 4],
            [2, 2, 2, 6],
            [6, 6, 4, 4],
            [2, 4, 4, 4]])

## 離散コサイン変換行列の定義

\begin{array}{l}
y[m]=\sqrt{\frac{2}{M}} k_{m} \sum_{n=0}^{N-1} \cos \left(\frac{(2 n+1) m \pi}{2 M}\right) x[n],\, m=0,1, \cdots, M-1 \\
x[n]=\sqrt{\frac{2}{M}} \sum_{m=0}^{N-1} k_{m} \cos \left(\frac{(2 n+1) m \pi}{2 M}\right) y[m],\, n=0,1, \cdots, M-1
\end{array}
ただし、
\begin{equation}
k_{m}=\left\{\begin{array}{cc}
\frac{1}{\sqrt{2}} & m=0 \\
1 & m=1,2, \cdots, M-1
\end{array}\right.
\end{equation}

### 順変換行列

```
dctSize = 4
arrayDct = np.matrix(np.zeros([dctSize,dctSize]))
for iCol in range(dctSize):
    for iRow in range(dctSize):
        k = 1
        if iRow == 0:
            k = 1/np.sqrt(2)
        arrayDct[iRow,iCol] = np.sqrt(2/dctSize)*k*np.cos((2*iCol+1)*iRow*np.pi/(2*dctSize))
display(arrayDct)

```

    matrix([[ 0.5       ,  0.5       ,  0.5       ,  0.5       ],
            [ 0.65328148,  0.27059805, -0.27059805, -0.65328148],
            [ 0.5       , -0.5       , -0.5       ,  0.5       ],
            [ 0.27059805, -0.65328148,  0.65328148, -0.27059805]])

### 逆変換行列

離散コサイン変換(DCT)は正規直交行列のため、逆変換行列は順変換行列の転置で与えられる。
\begin{equation}
\mathbf{C}^{-1}=\mathbf{C}^T
\end{equation}

```
arrayIdct = arrayDct.T
display(arrayIdct)
```

    matrix([[ 0.5       ,  0.65328148,  0.5       ,  0.27059805],
            [ 0.5       ,  0.27059805, -0.5       , -0.65328148],
            [ 0.5       , -0.27059805, -0.5       ,  0.65328148],
            [ 0.5       , -0.65328148,  0.5       , -0.27059805]])

## 二次元DCT
\begin{equation}
\mathbf{Y}=\mathbf{CXC}^T
\end{equation}

```
arrayY = np.dot(np.dot(arrayDct,arrayX),arrayDct.T)
display(arrayY)
```

    matrix([[ 1.50000000e+01, -1.14805030e+00, -9.99200722e-16,
             -2.77163860e+00],
            [-1.08239220e+00, -5.00000000e-01,  5.41196100e-01,
             -1.79289322e+00],
            [-1.00000000e+00, -3.82683432e-01, -2.00000000e+00,
             -9.23879533e-01],
            [ 2.61312593e+00,  3.20710678e+00, -1.30656296e+00,
             -5.00000000e-01]])

## 二次元逆DCT
\begin{equation}
\mathbf{X}=\mathbf{C}^{-1}\mathbf{YC}^{-T}
\end{equation}

```
arrayR = np.dot(np.dot(arrayIdct,arrayY),arrayIdct.T)
display(arrayR)
```

    matrix([[2., 6., 2., 4.],
            [2., 2., 2., 6.],
            [6., 6., 4., 4.],
            [2., 4., 4., 4.]])


# Interact Exercise 6

## Imports

Put the standard imports for Matplotlib, Numpy and the IPython widgets in the following cell.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

```python
from IPython.display import Image
from IPython.html.widgets import interact, interactive, fixed
```

    :0: FutureWarning: IPython widgets are experimental and may change in the future.

## Exploring the Fermi distribution

In quantum statistics, the [Fermi-Dirac](http://en.wikipedia.org/wiki/Fermi%E2%80%93Dirac_statistics) distribution is related to the probability that a particle will be in a quantum state with energy $\epsilon$. The equation for the distribution $F(\epsilon)$ is:

```python
Image('fermidist.png')
```

In this equation:

* $\epsilon$ is the single particle energy.
* $\mu$ is the chemical potential, which is related to the total number of particles.
* $k$ is the Boltzmann constant.
* $T$ is the temperature in Kelvin.

In the cell below, typeset this equation using LaTeX:

$$\begin{align}
F(\epsilon)=\frac1{e^{(\epsilon-\mu)/kT}+1}
\end{align}$$

Define a function `fermidist(energy, mu, kT)` that computes the distribution function for a given value of `energy`, chemical potential `mu` and temperature `kT`. Note here, `kT` is a single variable with units of energy. Make sure your function works with an array and don't use any `for` or `while` loops in your code.

```python
def fermidist(energy, mu, kT):
    """Compute the Fermi distribution at energy, mu and kT."""
    F=1/(np.exp((energy-mu)/kT)+1)
    return F
    raise NotImplementedError()
```

```python
assert np.allclose(fermidist(0.5, 1.0, 10.0), 0.51249739648421033)
assert np.allclose(fermidist(np.linspace(0.0,1.0,10), 1.0, 10.0),
    np.array([ 0.52497919,  0.5222076 ,  0.51943465,  0.5166605 ,  0.51388532,
               0.51110928,  0.50833256,  0.50555533,  0.50277775,  0.5       ]))
```

Write a function `plot_fermidist(mu, kT)` that plots the Fermi distribution $F(\epsilon)$ as a function of $\epsilon$ as a line plot for the parameters `mu` and `kT`.

* Use enegies over the range $[0,10.0]$ and a suitable number of points.
* Choose an appropriate x and y limit for your visualization.
* Label your x and y axis and the overall visualization.
* Customize your plot in 3 other ways to make it effective and beautiful.

```python
def plot_fermidist(mu, kT):
    energy=np.arange(10.0)
    y=fermidist(energy,mu,kT)
    plt.plot(energy, y)
    plt.xlabel('Energy,$\epsilon(J)$')
    plt.ylabel('Probability')
    plt.title('Fermi-Dirac Distribution')
    plt.ylim(0,1)
    plt.xlim(0,10)
```

```python
plot_fermidist(4.0, 1.0)
```

```python
assert True # leave this for grading the plot_fermidist function
```

Use `interact` with `plot_fermidist` to explore the distribution:

* For `mu` use a floating point slider over the range $[0.0,5.0]$.
* for `kT` use a floating point slider over the range $[0.1,10.0]$.

```python
interact(plot_fermidist,mu=(0,5.0),kT=(.1,10.0))

```

Provide complete sentence answers to the following questions in the cell below:

* What happens when the temperature $kT$ is low?
* What happens when the temperature $kT$ is high?
* What is the effect of changing the chemical potential $\mu$?
* The number of particles in the system are related to the area under this curve. How does the chemical potential affect the number of particles.

Use LaTeX to typeset any mathematical symbols in your answer.

When the temperature kT is low the probability of the energy being close to zero increases and the probability density falls rapidly as energy increases.
When kT is high the probability density becomes more evenly distributed among the energy levels.
Changing the chemical potential $\mu$ shifts the distribution up


# Support Vector Machines

Support Vector Machines (SVM) are an extension of the linear methods that attempt to separate classes with hyperplans.

These extensions come in three steps:
1. When classes are linearly separable, maximize the margin between the two classes
2. When classes are not linearly separable, maximize the margin but allow some samples within the margin. That is the soft margin
3. The "Kernel trick" to extend the separation to non linear frontieres

The boost in performance of the Kernel trick has made the SVM the best classification method of the 2000's until the deep neural nets.

### Learning goals

- Understand and implement SVM concepts stated above
- Reminder to the Lagrange multiplier and optimization theory
- Deal with a general purpose solver with constraints
- Apply SVM to a non linear problem (XOR) with a non linear kernel (G-RBF)

### References

- [1] [The Elements of Statistical Learning](https://web.stanford.edu/~hastie/ElemStatLearn/) - Trevor Hastie, Robert Tibshirani, Jerome Friedman, Springer
- [2] Convex Optimization - Stephen Boyd, Lieven Vandenberghe, Cambridge University Press
- [3] [Pattern Recognition and Machine Learning - Ch 7 demo](https://github.com/yiboyang/PRMLPY/blob/master/ch7/svm.py) - Christopher M Bishop, Github

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as pltcolors
from sklearn import linear_model, svm, discriminant_analysis, metrics
from scipy import optimize
import seaborn as sns
```

## Helpers

```python
def plotLine(ax, xRange, w, x0, label, color='grey', linestyle='-', alpha=1.):
    """ Plot a (separating) line given the normal vector (weights) and point of intercept """
    if type(x0) == int or type(x0) == float or type(x0) == np.float64:
        x0 = [0, -x0 / w[1]]
    yy = -(w[0] / w[1]) * (xRange - x0[0]) + x0[1]
    ax.plot(xRange, yy, color=color, label=label, linestyle=linestyle)

def plotSvm(X, y, support=None, w=None, intercept=0., label='Data', separatorLabel='Separator',
            ax=None, bound=[[-1., 1.], [-1., 1.]]):
    """ Plot the SVM separation, and margin """
    if ax is None:
        fig, ax = plt.subplots(1)

    im = ax.scatter(X[:,0], X[:,1], c=y, cmap=cmap, alpha=0.5, label=label)
    if support is not None:
        ax.scatter(support[:,0], support[:,1], label='Support', s=80, facecolors='none',
                   edgecolors='y', color='y')
        print("Number of support vectors = %d" % (len(support)))
    if w is not None:
        xx = np.array(bound[0])
        plotLine(ax, xx, w, intercept, separatorLabel)
        # Plot margin
        if support is not None:
            signedDist = np.matmul(support, w)
            margin = np.max(signedDist) - np.min(signedDist) * np.sqrt(np.dot(w, w))
            supportMaxNeg = support[np.argmin(signedDist)]
            plotLine(ax, xx, w, supportMaxNeg, 'Margin -', linestyle='-.', alpha=0.8)
            supportMaxPos = support[np.argmax(signedDist)]
            plotLine(ax, xx, w, supportMaxPos, 'Margin +', linestyle='--', alpha=0.8)
            ax.set_title('Margin = %.3f' % (margin))
    ax.legend(loc='upper left')
    ax.grid()
    ax.set_xlim(bound[0])
    ax.set_ylim(bound[1])
    cb = plt.colorbar(im, ax=ax)
    loc = np.arange(-1,1,1)
    cb.set_ticks(loc)
    cb.set_ticklabels(['-1','1'])
```

## The data model

Let's use a simple model with two Gaussians that are faraway in order to be separable

```python
colors = ['blue','red']
cmap = pltcolors.ListedColormap(colors)
nFeatures = 2
N = 100
```

```python
def generateBatchBipolar(n, mu=0.5, sigma=0.2):
    """ Two gaussian clouds on each side of the origin """
    X = np.random.normal(mu, sigma, (n, 2))
    yB = np.random.uniform(0, 1, n) > 0.5
    # y is in {-1, 1}
    y = 2. * yB - 1
    X *= y[:, np.newaxis]
    X -= X.mean(axis=0)
    return X, y
```

# 1. Maximum margin separator

The following explanation is about the binary classification but generalizes to more classes.

Let $X$ be the matrix of $n$ samples of the $p$ features. We want to separate the two classes of $y$ with an hyperplan (a straight line in 2D, that is $p=2$). The separation equation is:

$$ w^T x + b = 0, w \in \mathbb{R}^{p}, x \in \mathbb{R}^{p}, b \in \mathbb{R} $$

Given $x_0$ a point on the hyperplan, the signed distance of any point $x$ to the hyperplan is :
$$ \frac{w}{\Vert w \Vert} (x - x_0) = \frac{1}{\Vert w \Vert} (w^T x + b) $$

If $y$, such that $y \in \{-1, 1\}$, is the corresponding label of $x$, the (unsigned) distance is :
$$ \frac{y}{\Vert w \Vert} (w^T x + b) $$

This is the update quantity used by the Rosenblatt Perceptron.

The __Maximum margin separator__ is aiming at maximizing $M$ such that :
$$ \underset{w, b}{\max} M $$
__Subject to :__
- $y_i(x_i^T w + b) \ge M, i = 1..n$
- $\Vert w \Vert = 1$

$x_i$ and $y_i$ are samples of $x$ and $y$, a row of the matrix $X$ and the vector $y$.

However, we may change the condition on the norm of $w$ such that : $\Vert w \Vert = \frac 1M$

Leading to the equivalent statement of the maximum margin classifier :
$$ \min_{w, b} \frac 12 \Vert w \Vert^2 $$
__Subject to : $y_i(x_i^T w + b) \ge 1, i = 1..n$__

For more details, see [1, chap 4.5]

The corresponding Lagrange primal problem is :

$$\mathcal{L}_p(w, b, \alpha) = \frac 12 \Vert w \Vert^2 - \sum_{i=0}^n \alpha_i (y_i(x_i^T w + b) - 1)$$

__Subject to:__
- $\alpha_i \ge 0, i\in 1..n$

This shall be __minimized__ on $w$ and $b$, using the corresponding partial derivates equal to 0, we get :
$$\begin{align}
\sum_{i=0}^n \alpha_i y_i x_i &= w \\
\sum_{i=0}^n \alpha_i y_i &= 0
\end{align}$$

From $\mathcal{L}_p$, we get the (Wolfe) dual :
$$\begin{align}
\mathcal{L}_d (\alpha)
&= \sum_{i=0}^n \alpha_i - \frac 12 \sum_{i=0}^n \sum_{k=0}^n \alpha_i \alpha_k y_i y_k x_i^T x_k \\
&= \sum_{i=0}^n \alpha_i - \frac 12 \sum_{i=0}^n \sum_{k=0}^n \langle \alpha_i y_i x_i, \alpha_k y_k  x_k \rangle \\
\end{align}$$

__Subject to :__
- $\alpha_i \ge 0, i\in 1..n$
- $\sum_{i=0}^n \alpha_i y_i = 0$

Which is a concave problem that is __maximized__ using a solver.

Strong duality requires (KKT) [2, chap. 5.5]:
- $\alpha_i (y_i(x_i^T w + b) - 1) = 0,  \forall i \in 1..n$

Implying that :
- If $\alpha_i > 0$, then $y_i(x_i^T w + b) = 1$, meaning that $x_i$ is on one of the two hyperplans located at the margin distance from the separating hyperplan. $x_i$ is said to be a support vector
- If $y_i(x_i^T w + b) > 1$, the distance of $x_i$ to the hyperplan is larger than the margin.

### Train data

To demonstrate the maximum margin classifier, a dataset with separable classes is required. Let's use a mixture of two gaussian distributed classes with mean and variance such that the two classes are separated.

```python
xTrain0, yTrain0 = generateBatchBipolar(N, sigma=0.2)
plotSvm(xTrain0, yTrain0)
```

## Implementation of the Maximum margin separator

$$\mathcal{L}_d = \sum_{i=0}^n \alpha_i - \frac 12 \sum_{i=0}^n \sum_{k=0}^n \alpha_i \alpha_k y_i y_k x_i^T x_k $$

__Subject to :__
- $\sum_{i=0}^n \alpha_i y_i = \langle \alpha, y \rangle = 0$
- $\alpha_i \ge 0, i\in 1..n$

The classifier is built on the scipy.optimize.minimum solver. The implementation is correct but inefficient as it is not taking into account for the sparsity of the $\alpha$ vector.

```python
class MaxMarginClassifier:

    def __init__(self):
        self.alpha = None
        self.w = None
        self.supportVectors = None

    def fit(self, X, y):
        N = len(y)
        # Gram matrix of (X.y)
        Xy = X * y[:, np.newaxis]
        GramXy = np.matmul(Xy, Xy.T)

        # Lagrange dual problem
        def Ld0(G, alpha):
            return alpha.sum() - 0.5 * alpha.dot(alpha.dot(G))

        # Partial derivate of Ld on alpha
        def Ld0dAlpha(G, alpha):
            return np.ones_like(alpha) - alpha.dot(G)

        # Constraints on alpha of the shape :
        # -  d - C*alpha  = 0
        # -  b - A*alpha >= 0
        A = -np.eye(N)
        b = np.zeros(N)
        constraints = ({'type': 'eq',   'fun': lambda a: np.dot(a, y), 'jac': lambda a: y},
                       {'type': 'ineq', 'fun': lambda a: b - np.dot(A, a), 'jac': lambda a: -A})

        # Maximize by minimizing the opposite
        optRes = optimize.minimize(fun=lambda a: -Ld0(GramXy, a),
                                   x0=np.ones(N),
                                   method='SLSQP',
                                   jac=lambda a: -Ld0dAlpha(GramXy, a),
                                   constraints=constraints)
        self.alpha = optRes.x
        self.w = np.sum((self.alpha[:, np.newaxis] * Xy), axis=0)
        epsilon = 1e-6
        self.supportVectors = X[self.alpha > epsilon]
        # Any support vector is at a distance of 1 to the separation plan
        # => use support vector #0 to compute the intercept, assume label is in {-1, 1}
        supportLabels = y[self.alpha > epsilon]
        self.intercept = supportLabels[0] - np.matmul(self.supportVectors[0].T, self.w)

    def predict(self, X):
        """ Predict y value in {-1, 1} """
        assert(self.w is not None)
        assert(self.w.shape[0] == X.shape[1])
        return 2 * (np.matmul(X, self.w) > 0) - 1
```

Reference:
- https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize

```python
model00 = MaxMarginClassifier()
model00.fit(xTrain0, yTrain0)
model00.w, model00.intercept
```

    (array([2.59368501, 2.31755533]), -0.23352398150198717)

```python
fig, ax = plt.subplots(1, figsize=(12, 7))
plotSvm(xTrain0, yTrain0, model00.supportVectors, model00.w, model00.intercept, label='Training', ax=ax)
```

## Maximum margin classifier using Scikit Learn (SVC)

SVC is used in place of LinearSVC as the support vectors are provided. These vectors are displayed in the graph here below.

Set a high $C$ parameter to disable soft margin

```python
model01 = svm.SVC(kernel='linear', gamma='auto', C = 1e6)
model01.fit(xTrain0, yTrain0)
model01.coef_[0], model01.intercept_[0]
```

    (array([2.59448981, 2.31753528]), -0.23340120915728754)

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTrain0, yTrain0, model01.support_vectors_, model01.coef_[0], model01.intercept_[0],
        label='Training', ax=ax)
```

The two implementations of the linear SVM agree on the ceofficients and margin. Good !

### Comparison of the maximum margin classifier to the Logistic regression and Linear Discriminant Analysis (LDA)

Logistic regression is based on the linear regression that is the computation of the square error of any point $x$ to the separation plan and a projection on the probability space using the sigmoid in order to compute the binary cross entropy, see ([HTML](ClassificationContinuous2Features.html) / [Jupyter](ClassificationContinuous2Features.ipynb)).

LDA is assuming a Gaussian mixture prior (our case) and performs bayesian inference.

```python
model02 = linear_model.LogisticRegression(solver='lbfgs')
model02.fit(xTrain0, yTrain0)
model02.coef_[0], model02.intercept_[0]
```

    (array([2.66226303, 3.03715145]), 0.18691381097527718)

```python
model03 = discriminant_analysis.LinearDiscriminantAnalysis(solver='svd')
model03.fit(xTrain0, yTrain0)
model03.coef_[0], model03.intercept_[0]
```

    (array([21.42974081, 29.24074964]), 1.626159053899352)

We observe that the coefficients of the three models are very different in amplitude but globally draw a separator line with slope $-\frac \pi4$ in the 2D plan

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTrain0, yTrain0, w=model01.coef_[0], intercept=model01.intercept_[0],
        separatorLabel='Max Margin SVM', label='Training', ax=ax)
xx = np.array([-1., 1.])
plotLine(ax, xx, w=model02.coef_[0], x0=model02.intercept_[0], label='Logistic', color='g')
plotLine(ax, xx, w=model03.coef_[0], x0=model03.intercept_[0], label='LDA', color='c')
ax.legend();
```

# 2. Soft Margin Linear SVM for non separable classes

The example above has little interest as the separation is trivial.

Using the same SVM implementation on a non separable case would not be possible, the solver would fail.

Here comes the soft margin: some $x_i$ are allowed to lie in between the two margin bars.

The __Soft margin linear SVM__ is adding a regularization parameter in maximizing $M$:

$$ \underset{w, b}{\max} M ( 1 - \xi_i) $$
__Subject to $\forall i = 1..n$:__
- $y_i(x_i^T w + b) \ge M$
- $\Vert w \Vert = 1$
- $\xi_i \ge 0$

Equivalently :
$$ \min_{w, b} \frac 12 \Vert w \Vert^2 + C \sum_{i=1}^n \xi_i$$
__Subject to $\forall i = 1..n$:__
- $\xi_i \ge 0$
- $y_i(x_i^T w + b) \ge 1 - \xi_i$

The corresponding Lagrange primal problem is :

$$\mathcal{L}_p(w, b, \alpha, \mu) = \frac 12 \Vert w \Vert^2 - \sum_{i=0}^n \alpha_i (y_i(x_i^T w + b) - (1 - \xi_i) - \sum_{i=0}^n \mu_i \xi_i  $$

__Subject to $\forall i\in 1..n$:__
- $\alpha_i \ge 0$
- $\mu_i \ge 0$
- $\xi_i \ge 0$

This shall be minimized on $w$, $b$ and $\xi_i$, using the corresponding partial derivates equal to 0, we get :
$$\begin{align}
\sum_{i=0}^n \alpha_i y_i x_i &= w \\
\sum_{i=0}^n \alpha_i y_i &= 0 \\
\alpha_i &= C - \mu_i
\end{align}$$

From $\mathcal{L}_p$, we get the (Wolfe) dual :
$$\begin{align}
\mathcal{L}_d (\alpha)
&= \sum_{i=0}^n \alpha_i - \frac 12 \sum_{i=0}^n \sum_{k=0}^n \alpha_i \alpha_k y_i y_k x_i^T x_k \\
&= \sum_{i=0}^n \alpha_i - \frac 12 \sum_{i=0}^n \sum_{k=0}^n \langle \alpha_i y_i x_i, \alpha_k y_k  x_k \rangle \\
\end{align}$$

__Subject to $\forall i\in 1..n$:__
- $0 \le \alpha_i \le C$
- $\sum_{i=0}^n \alpha_i y_i = 0$

This problem is very similar to the one of the Maximum margin separator, but with one more constraint on $\alpha$.

It is a concave problem that is maximized using a solver.

Extra conditions to get strong duality are required (KKT), $\forall i \in 1..n$:
- $\alpha_i (y_i(x_i^T w + b) - (1 - \xi_i)) = 0$
- $\mu_i \xi_i = 0$
- $y_i(x_i^T w + b) - (1 - \xi_i) \ge 0$

More detailed explainations are in [1 chap. 12.1, 12.2]

## Data model

Let's reuse the same model made of two gaussians, but with larger variance in order to mix the positive and negative points

```python
xTrain1, yTrain1 = generateBatchBipolar(N, mu=0.3, sigma=0.3)
plotSvm(xTrain1, yTrain1, label='Training')
```

## Custom implementation

Changes to the Maximum margin classifier are identified by "# <---"

```python
class LinearSvmClassifier:

    def __init__(self, C):
        self.C = C                                 # <---
        self.alpha = None
        self.w = None
        self.supportVectors = None

    def fit(self, X, y):
        N = len(y)
        # Gram matrix of (X.y)
        Xy = X * y[:, np.newaxis]
        GramXy = np.matmul(Xy, Xy.T)

        # Lagrange dual problem
        def Ld0(G, alpha):
            return alpha.sum() - 0.5 * alpha.dot(alpha.dot(G))

        # Partial derivate of Ld on alpha
        def Ld0dAlpha(G, alpha):
            return np.ones_like(alpha) - alpha.dot(G)

        # Constraints on alpha of the shape :
        # -  d - C*alpha  = 0
        # -  b - A*alpha >= 0
        A = np.vstack((-np.eye(N), np.eye(N)))             # <---
        b = np.hstack((np.zeros(N), self.C * np.ones(N)))  # <---
        constraints = ({'type': 'eq',   'fun': lambda a: np.dot(a, y), 'jac': lambda a: y},
                       {'type': 'ineq', 'fun': lambda a: b - np.dot(A, a), 'jac': lambda a: -A})

        # Maximize by minimizing the opposite
        optRes = optimize.minimize(fun=lambda a: -Ld0(GramXy, a),
                                   x0=np.ones(N),
                                   method='SLSQP',
                                   jac=lambda a: -Ld0dAlpha(GramXy, a),
                                   constraints=constraints)
        self.alpha = optRes.x
        self.w = np.sum((self.alpha[:, np.newaxis] * Xy), axis=0)
        epsilon = 1e-6
        self.supportVectors = X[self.alpha > epsilon]
        # Support vectors is at a distance <= 1 to the separation plan
        # => use min support vector to compute the intercept, assume label is in {-1, 1}
        signedDist = np.matmul(self.supportVectors, self.w)
        minDistArg = np.argmin(signedDist)
        supportLabels = y[self.alpha > epsilon]
        self.intercept = supportLabels[minDistArg] - signedDist[minDistArg]

    def predict(self, X):
        """ Predict y value in {-1, 1} """
        assert(self.w is not None)
        assert(self.w.shape[0] == X.shape[1])
        return 2 * (np.matmul(X, self.w) > 0) - 1
```

```python
model10 = LinearSvmClassifier(C=1)
model10.fit(xTrain1, yTrain1)
model10.w, model10.intercept
```

    (array([2.40325078, 1.96349807]), 0.07480337641501511)

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTrain1, yTrain1, model10.supportVectors, model10.w, model10.intercept, label='Training', ax=ax)
```

### Linear SVM using Scikit Learn

```python
model11 = svm.SVC(kernel='linear', gamma='auto', C = 1)
model11.fit(xTrain1, yTrain1)
model11.coef_[0], model11.intercept_[0]
```

    (array([2.40564683, 1.94590186]), 0.07537008170330341)

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTrain1, yTrain1, model11.support_vectors_, model11.coef_[0], model11.intercept_[0],
        label='Training', ax=ax)
```

With the soft margin, the support vectors are all the vectors on the boundary or within the margin slab.

The custom and SKLearn implementations are matching !

### Comparison of the soft margin classifier to the Logistic regression and Linear Discriminant Analysis (LDA)

```python
model12 = linear_model.LogisticRegression(solver='lbfgs')
model12.fit(xTrain1, yTrain1)
model12.coef_[0], model12.intercept_[0]
```

    (array([2.82950189, 2.54532752]), 0.1166247961098039)

```python
model13 = discriminant_analysis.LinearDiscriminantAnalysis(solver='svd')
model13.fit(xTrain1, yTrain1)
model13.coef_[0], model13.intercept_[0]
```

    (array([6.24615021, 5.35630463]), 0.1095246818198905)

As shown below, the three models separator hyperplans are very similar, negative slope.

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTrain1, yTrain1, w=model11.coef_[0], intercept=model11.intercept_[0], label='Training',
        separatorLabel='Soft Margin SVM', ax=ax)
xx = np.array([-1., 1.])
plotLine(ax, xx, w=model12.coef_[0], x0=model12.intercept_[0], label='Logistic reg', color='orange')
plotLine(ax, xx, w=model13.coef_[0], x0=model13.intercept_[0], label='LDA', color='c')
ax.legend();
```

### Validation with test data

```python
xTest1, yTest1 = generateBatchBipolar(2*N, mu=0.3, sigma=0.3)
```

#### Helpers for binary classification performance

```python
def plotHeatMap(X, classes, title=None, fmt='.2g', ax=None, xlabel=None, ylabel=None):
    """ Fix heatmap plot from Seaborn with pyplot 3.1.0, 3.1.1
        https://stackoverflow.com/questions/56942670/matplotlib-seaborn-first-and-last-row-cut-in-half-of-heatmap-plot
    """
    ax = sns.heatmap(X, xticklabels=classes, yticklabels=classes, annot=True, \
                     fmt=fmt, cmap=plt.cm.Blues, ax=ax) #notation: "annot" not "annote"
    bottom, top = ax.get_ylim()
    ax.set_ylim(bottom + 0.5, top - 0.5)
    if title:
        ax.set_title(title)
    if xlabel:
        ax.set_xlabel(xlabel)
    if ylabel:
        ax.set_ylabel(ylabel)

def plotConfusionMatrix(yTrue, yEst, classes, title=None, fmt='.2g', ax=None):
    plotHeatMap(metrics.confusion_matrix(yTrue, yEst), classes, title, fmt, ax, xlabel='Estimations', \
                ylabel='True values');
```

### Confusion matrices

```python
fig, axes = plt.subplots(1, 3, figsize=(16, 3))
for model, ax, title in zip([model10, model12, model13], axes, ['Custom linear SVM', 'Logistic reg', 'LDA']):
    yEst = model.predict(xTest1)
    plotConfusionMatrix(yTest1, yEst, colors, title, ax=ax)
```

There is no clear winner, all models are performing equally well.

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTest1, yTest1, w=model10.w, intercept=model10.intercept, separatorLabel='Cust. linear SVM', ax=ax)
xx = np.array([-1., 1.])
plotLine(ax, xx, w=model12.coef_[0], x0=model12.intercept_[0], label='Logistic reg', color='orange')
plotLine(ax, xx, w=model13.coef_[0], x0=model13.intercept_[0], label='LDA', color='c')
ax.legend();
```

# 3. The "kernel trick" for non linearly separable classes

Let's use a very famous dataset showing the main limitation of the Logistic regression and LDA : the XOR.

```python
def generateBatchXor(n, mu=0.5, sigma=0.5):
    """ Four gaussian clouds in a Xor fashion """
    X = np.random.normal(mu, sigma, (n, 2))
    yB0 = np.random.uniform(0, 1, n) > 0.5
    yB1 = np.random.uniform(0, 1, n) > 0.5
    # y is in {-1, 1}
    y0 = 2. * yB0 - 1
    y1 = 2. * yB1 - 1
    X[:,0] *= y0
    X[:,1] *= y1
    X -= X.mean(axis=0)
    return X, y0*y1
```

```python
xTrain3, yTrain3 = generateBatchXor(2*N, sigma=0.25)
plotSvm(xTrain3, yTrain3)
xTest3, yTest3 = generateBatchXor(2*N, sigma=0.25)
```

## Logistic regression and LDA on XOR problem

```python
model32 = linear_model.LogisticRegression(solver='lbfgs')
model32.fit(xTrain3, yTrain3)
model32.coef_[0], model32.intercept_[0]
```

    (array([-0.0486223,  0.0060628]), -0.04001287348502901)

```python
model33 = discriminant_analysis.LinearDiscriminantAnalysis(solver='svd')
model33.fit(xTrain3, yTrain3)
model33.coef_[0], model33.intercept_[0]
```

    (array([-0.05117155,  0.00630136]), -0.040013903109986315)

The linear separators are sometimes mitigating the issue by isolating a single class within a corner. Or they are simply fully failing (separator is of limit).

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTrain3, yTrain3, w=model32.coef_[0], intercept=model32.intercept_[0], label='Training',
        separatorLabel='Logistic reg', ax=ax)
xx = np.array([-1., 1.])
plotLine(ax, xx, w=model33.coef_[0], x0=model33.intercept_[0], label='LDA', color='c')
ax.legend();
```

## Introducing the Kernel trick

When using linear separators like the regression, the traditional way to deal with non linear functions is to expand the feature space using powers and products of the initial features. This is also necessary in case of multiclass problems as shown in [1 chap. 4.2].

There are limits to this trick. For example, the XOR problem is not handled proprely.

The SVM has used a new method known as the "Kernel trick".

Let's apply a transformation to $x$ using function $h(x)$.

The Lagrange (Wolfe) dual problem becomes :
$$\begin{align}
\mathcal{L}_d (\alpha)
&= \sum_{i=0}^n \alpha_i - \frac 12 \sum_{i=0}^n \sum_{k=0}^n \alpha_i \alpha_k y_i y_k h(x_i)^T h(x_k) \\
&= \sum_{i=0}^n \alpha_i - \frac 12 \sum_{i=0}^n \sum_{k=0}^n \alpha_i \alpha_k \langle y_i h(x_i), y_k  h(x_k) \rangle \\
\end{align}$$

__Subject to $\forall i\in 1..n$:__
- $0 \le \alpha_i \le C$
- $\sum_{i=0}^n \alpha_i y_i = 0$

Since $ w = \sum_{i=0}^n \alpha_i y_i h(x_i)$, the prediction function is now :
$$ f(x) = sign(w^T h(x) + b) = sign \left(\sum_{i=0}^n \alpha_i y_i \langle h(x_i), h(x) \rangle \right) $$

This prediction needs to be computed for $\alpha_i > 0$, that are support vectors.

Both the fit and prediction are based on the inner product $K(x, x') = \langle h(x), h(x') \rangle$, also known as the kernel function. This function shall be symmetric, semi-definite.

Popular kernel is the Gaussian Radial Basis Function (RBF) : $K(x, x') = exp(- \gamma \Vert x - x' \Vert^2 )$

### Custom implementation of the SVM with G-RBF kernel

Modifications made on the Linear SVM implementation are enclosed in blocks starting with _"# --->"_ and ending with _"# <---"_

```python
class KernelSvmClassifier:

    def __init__(self, C, kernel):
        self.C = C
        self.kernel = kernel          # <---
        self.alpha = None
        self.supportVectors = None

    def fit(self, X, y):
        N = len(y)
        # --->
        # Gram matrix of h(x) y
        hXX = np.apply_along_axis(lambda x1 : np.apply_along_axis(lambda x2:  self.kernel(x1, x2), 1, X),
                                  1, X)
        yp = y.reshape(-1, 1)
        GramHXy = hXX * np.matmul(yp, yp.T)
        # <---

        # Lagrange dual problem
        def Ld0(G, alpha):
            return alpha.sum() - 0.5 * alpha.dot(alpha.dot(G))

        # Partial derivate of Ld on alpha
        def Ld0dAlpha(G, alpha):
            return np.ones_like(alpha) - alpha.dot(G)

        # Constraints on alpha of the shape :
        # -  d - C*alpha  = 0
        # -  b - A*alpha >= 0
        A = np.vstack((-np.eye(N), np.eye(N)))             # <---
        b = np.hstack((np.zeros(N), self.C * np.ones(N)))  # <---
        constraints = ({'type': 'eq',   'fun': lambda a: np.dot(a, y),     'jac': lambda a: y},
                       {'type': 'ineq', 'fun': lambda a: b - np.dot(A, a), 'jac': lambda a: -A})

        # Maximize by minimizing the opposite
        optRes = optimize.minimize(fun=lambda a: -Ld0(GramHXy, a),
                                   x0=np.ones(N),
                                   method='SLSQP',
                                   jac=lambda a: -Ld0dAlpha(GramHXy, a),
                                   constraints=constraints)
        self.alpha = optRes.x
        # --->
        epsilon = 1e-8
        supportIndices = self.alpha > epsilon
        self.supportVectors = X[supportIndices]
        self.supportAlphaY = y[supportIndices] * self.alpha[supportIndices]
        # <---

    def predict(self, X):
        """ Predict y values in {-1, 1} """
        # --->
        def predict1(x):
            x1 = np.apply_along_axis(lambda s: self.kernel(s, x), 1, self.supportVectors)
            x2 = x1 * self.supportAlphaY
            return np.sum(x2)

        d = np.apply_along_axis(predict1, 1, X)
        return 2 * (d > 0) - 1
        # <---
```

```python
def GRBF(x1, x2):
    diff = x1 - x2
    return np.exp(-np.dot(diff, diff) * len(x1) / 2)

model30 = KernelSvmClassifier(C=5, kernel=GRBF)
model30.fit(xTrain3, yTrain3)
```

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTrain3, yTrain3, support=model30.supportVectors, label='Training', ax=ax)

# Estimate and plot decision boundary
xx = np.linspace(-1, 1, 50)
X0, X1 = np.meshgrid(xx, xx)
xy = np.vstack([X0.ravel(), X1.ravel()]).T
Y30 = model30.predict(xy).reshape(X0.shape)
ax.contour(X0, X1, Y30, colors='k', levels=[-1, 0], alpha=0.3, linestyles=['-.', '-']);
```

## Scikit Learn SVM with Radial basis kernel

```python
model31 = svm.SVC(kernel='rbf', C=10, gamma=1/2, shrinking=False)
model31.fit(xTrain3, yTrain3);
```

```python
fig, ax = plt.subplots(1, figsize=(11, 7))
plotSvm(xTrain3, yTrain3, support=model31.support_vectors_, label='Training', ax=ax)

# Estimate and plot decision boundary
Y31 = model31.predict(xy).reshape(X0.shape)
ax.contour(X0, X1, Y31, colors='k', levels=[-1, 0], alpha=0.3, linestyles=['-.', '-']);
```

### SVM with RBF performance on XOR

```python
fig, axes = plt.subplots(1, 2, figsize=(11, 3))
for model, ax, title in zip([model30, model31], axes, ["Custom SVM with RBF", "SKLearn SVM with RBF"]):
    yEst3 = model.predict(xTest3)
    plotConfusionMatrix(yTest3, yEst3, colors, title, ax=ax)
```

Both models' predictions are almost matching on the XOR example.

## Conclusion

We have shown the power of SVM classifiers for non linearly separable problems. From the end of the 1990's, SVM was the leading machine learning algorithm family for many problems. This situation has changed a little since 2010 as deep learning has shown better performance for some classes of problems. However, SVM remains stronger in many contexts. For example, the amount of training data for SVM is lower than the one required for deep learning.

### Where to go from here

- Multiclass classifier using Neural Nets in Keras ([HTML](ClassificationMulti2Features-Keras.html) / [Jupyter](ClassificationMulti2Features-Keras.ipynb))
- Multiclass classifier using Decision Trees ([HTML](ClassificationMulti2Features-Tree.html) / [Jupyter](ClassificationMulti2Features-Tree.ipynb))
- Bivariate continuous function approximation with Linear Regression ([HTML](ClassificationContinuous2Features.html) / [Jupyter](ClassificationContinuous2Features.ipynb))
- Bivariate continuous function approximation with k Nearest Neighbors ([HTML](ClassificationContinuous2Features-KNN.html) / [Jupyter](ClassificationContinuous2Features-KNN.ipynb))


# Lecture 19:  Numerical Solutions to the Diffusion Equation  (Spectral Method)

## What to Learn?

* The assumptions of the spectral method
* The possible choices for basis vectors
* The linear independence of resulting ODEs
* One way to code the FFT/IFFT
* Finite differencing of the amplitudes
* Useful data structures and pointer swapping using tuples

## What to do?

* Solve the diffusion equation using a spectral method.
* Analyze two example implementations and determine the difference scheme.

### Introduction
----

In this course the term _spectral method_ refers to a numerical solution composed of a finite set of basis functions and the time evolution of their amplitudes.  Practically, the choice of basis functions will satisfy the boundary conditions and the initial amplitudes will satisfy the initial conditions.  The evolution of the amplitudes can then be written as a finite difference in time.  Although other choices for basis functions are [possible](./images/Boyd_Table.png)$^*$, we will focus on Fourier functions.

$*$ Chebyshev & Fourier Spectral Methods, Boyd, John P., Springer 1989

### Developing the Logic of the Spectral Method
----

The spectral method assumes that the solution to a PDE (the function $c(x,t)$) can be represented as a series expansion that contains time dependent amplitudes (the $a_k(t)$ factors) and spatially varying functions (the $\phi_k(x)$ terms are unit Fourier vectors).  Note the similarity to the approach taken in the separation of variables method in the following discussion.

To begin, we write:

$$
c(x,t) = \sum_{k=0}^N a_k(t)\phi_k(x)
$$

Choosing the unit vectors from an orthonormal set (like Fourier) permits decoupling the summation into $N$ independent equations that can be solved simultaneously.  Furthermore, the assumption of the orthonormal set allows the time dependence to be placed within the amplitudes so that the independent variables are seperated.  Therefore we refer to the $a_k(t)$ as the set of amplitudes with one $a_k$ for each of the $k$-basis vectors, $\phi_k(x)$.  There is no restriction on the value of $N$ although practical considerations that relate to diffusive problems and the "smoothness" of the solutions will require just a few $N$ terms to achieve acceptable accuracy.  The form of $\phi$ depends on the problem being solved; the boundary conditions and the initial conditions will affect this choice.

Our example problem:

$$
\frac{\partial c(x,t)}{\partial t} = \frac{\partial^2 c(x,t)}{\partial x^2}
$$

with the initial and boundary conditions:

$$
c(0,t) = 0\\
c(L,t) = 0\\
c(x,0) = c_0(x)
$$

The problem requires that our boundary values for the $c(x,t)$ at $0$ and $L$ are zero.  One possible choice is a series of $\sin$ terms.  This satisfies the boundary conditions and enforces perodicity of the solution.

$$
\phi_k(x) = \sin \left( \frac{k\pi x}{L} \right)
$$

Begin by importing relevant libraries and defining the symbols we expect to use.

```python
import sympy as sp
x, y, z, t = sp.symbols('x y z t')
k, m, n = sp.symbols('k m n', integer=True)
f, g, h = sp.symbols('f g h', cls=sp.Function)
sp.var('a_k, phi, c', cls=sp.Function);
sp.var('L', real=True);
sp.init_printing();
```

The assumed form of the solution is:

```python
elementK = sp.Eq(c(x,t),a_k(t)*sp.sin(k*sp.pi*x/L))
elementK
```

```python
sp.acos()
```

Proceed by substituting the series expansion into the PDE and performing the differentiations as defined:

```python
spaceDeriv = elementK.rhs.diff(x,2)
spaceDeriv
```

```python
timeDeriv = elementK.rhs.diff(t,1)
timeDeriv
```

Our final differential equation represented in $a(t)$ is therefore:

$$
\sum_{k=0}^N \sin{\left (\frac{\pi x}{L} k \right )} \frac{d a_k{\left (t \right )}}{d t}  = - \sum_{k=0}^N \frac{\pi^{2} k^{2}}{L^{2}} a_k{\left (t \right )} \sin{\left (\frac{\pi x}{L} k \right )}
$$

As a reminder, a $\sin$ series is orthogonal over $0 < x < 2\pi$ if the following integral is zero:

```python
# m and n are symbols defined as integers
sinIntegral = sp.Integral(sp.sin(n*x)*sp.sin(m*x),(x,0,2*sp.pi))
sinIntegral
```

```python
sinIntegral.doit()
```

Because the integral is zero for $m \neq n$, then the series on the LHS:

$$
\sum_{k=0}^N \sin{\left (\frac{\pi x}{L} k \right )} \frac{d a_k{\left (t \right )}}{d t}
$$

is a linear system.  This is also true for the RHS:

$$
- \sum_{k=0}^N \frac{\pi^{2} k^{2}}{L^{2}} a_k{\left (t \right )} \sin{\left (\frac{\pi x}{L} k \right )}
$$

The principle of superposition permits us to split this summation into N independent ordinary differential equations, solve each one, and then sum the solutions to produce the answer to the original PDE.  To continue developing a solution for these N independent ODEs it is necessary to analyze the amplitude ODE and define a differencing scheme.  This will be illustrated using SymPy:

```python
ai, aip1 = sp.symbols('a^{i}_k, a^{i+1}_k')
dt = sp.Symbol(r'\Delta t')

differenceEquation = sp.Eq((ai-aip1)/dt,((sp.pi**2*k**2*ai)/L**2))
differenceEquation
```

```python
odeSolution = sp.solveset(differenceEquation,aip1)
odeSolution
```

The above solution results in the following difference scheme:

$$
a^{i+1}_k = a^{i}_k \left( 1 - \frac{\pi^{2} k^{2}}{L^{2}} \Delta t \right)
$$

The timestep, $dt$, should be chosen small enough such that the $a_k$ decay at each timestep.

### Implementation of the Spectral Method
----

An annotated implementation of the spectral method is developed in the next section.  Unlike the development above, we use the full Fourier series for the basis functions.  This is a basic implementation that could be improved with the addition of helper and visualization functions.  `SciPy` provides $\sin$ and $\cos$ transforms for other boundary conditions where Fourier may be inappropriate.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
```

The following quantities are defined:

* `numPoints`: the number of points in the grid
* `L`: the total length of the domain
* `dt`:  the numerical timestep
* `steps`: the number of timesteps to compute
* `x`: a vector containing the x-position for each grid point in the domain from $0$ to $L$ with `numPoints` entries in the vector.

```python
numPoints = 20
L = 1.0
dt = 0.0001
steps = 1000
# we have access to np.pi for $\pi$
x = np.linspace(0,L,numPoints)
```

The following vectors are defined:

* `c_new` will hold the $c(x,t)$ values at the start and end of the numerical computation
* `a_old` will hold the amplitudes at the current timestep (i.e. the $a^{i}_k$)
* `a_new` will hold the amplitudes at the next timestep (i.e. the $a^{i+1}_k$)

They are initalized to zero at the start of the calculation.  Each of these vectors is the same shape as `x` to hold the results of the FFT and IFFT.  They are declared as `complex128` to accomodate different initial conditions where the FFT would produce complex valued results.

```python
c_new = np.zeros((numPoints), dtype='complex128')
a_old = np.zeros((numPoints), dtype='complex128')
a_new = np.zeros((numPoints), dtype='complex128')
```

In previous example calculations a set of basis vectors was created to illustrate Fourier transforms graphically and the terms in a Fourier series.  That is not part of this calculation, however this could be easily added if visualizing the results in more detail is desireable.

The difference equation requires values for the square of the Fourier numbers, $k^2$:

$$
a^{i+1}_k = a^{i}_k \left( 1 - \frac{\pi^{2} k^{2}}{L^{2}} \Delta t \right)
$$

To ensure appropriate matching between the $a_k(t)$ and $k$ the helper function `fftfreq()` is used.  Using `fftfreq` with `fft` and `ifft` ensures that the bookeeping of amplitudes and $k$ values is done correctly.  In the equation for the time rate of change in the amplitudes there is a constant $k^2$ term - computing this first avoids having to repeatedly compute the quantity each time step.

`NumPy` arrays are called by reference so it is necessary to perform an element-by-element "deep copy" of the data from one array into another array when building the initial condition. The helper function `np.copyto` provides this capability.  `a_new` is then filled with the amplitudes corresponding to the initial condition.

```python
k = np.fft.fftfreq(numPoints, d=L/(numPoints-1))
k2 = k**2
initialCondition = np.sin(np.pi*x/L)

# create an initial condition (this could be a simple function like x**2)
np.copyto(c_new, initialCondition)
# transform it (dft or sin transform)
np.copyto(a_new,np.fft.fft(c_new))
```

Instabilities will occur if the amplitudes do not decay at each timestep.  The problem is that the condition depends on the wavenumber - so a suitable $\Delta t$ must be chosen that satisfies the most restrictive condition for the largest wavenumber.  Using a Boolean expression it is possible to check to see if all of the wavenumbers result in a factor less than one:

```python
(dt*np.pi**2*k2)/L**2 < 1
```

    array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
            True,  True,  True,  True,  True,  True,  True,  True,  True,
            True,  True])

If any the results are `False` then the numerical calculation will not converge, if they are all `True` then it is possible to complete the numerical calculation.  The next code block performs the numerical iterations of the amplitudes.  First the pointers to `a_new` and `a_old` are swapped and then `a_new` is filled with the new values based on the `a_old` values.  This sequence of operations is performed for the number of timesteps given in `steps`:

```python
for i in range(steps):
    # swap pointers
    a_new, a_old = a_old, a_new
    # find new amplitudes
    np.copyto(a_new, a_old*(1-(dt*np.pi**2*k2)/L**2))
```

When the requested number of steps have been computed, we use the inverse Fourier transform to compute the concentration profile and store those results in `c_new`:

```python
# inverse transform it
np.copyto(c_new, np.fft.ifft(a_new))
```

After the computation, the concentration profile are displayed with a helper function:

```python
def makePlot():
    fig = plt.figure()
    axes = fig.add_axes([0.1, 0.1, 0.8, 0.8]) # left, bottom, width, height (range 0 to 1)
    axes.plot(x, c_new.real, 'r')
    # Setting the y-limit cleans up the plot.
    axes.set_ylim([0.0,1.0])
    axes.set_xlabel('Distance $x$')
    axes.set_ylabel('Concentration $c(x,t)$')
    axes.set_title('Concentration Profile solved by Spectral Method')
    plt.show()
    return
```

```python
makePlot()
```

Another solution scheme could preserve the values of the concentration.  I choose not to do that here for simplicity.  Doing so would permit a small animation of the diffusion process, however.  This is left to the student to implement.

### Reading Assignments and Practice
----

Analyze the spectral difference schemes in the following two examples.  For reference and attribution the codes can be found [here](https://open.umich.edu/find/open-educational-resources/literature-science-arts/parallel-spectral-numerical-methods).  The materials are provided under a Creative Commons license with attribution to the original authors whose names can be found at the above link.

```python
# %load Heat_Eq_1D_Spectral_BE.py
#!/usr/bin/env python
"""
Solving Heat Equation using pseudospectral methods with Backwards Euler:
u_t= \alpha*u_xx
BC = u(0)=0 and u(2*pi)=0 (Periodic)
IC=sin(x)
"""

import math
import numpy
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator

# Grid
N = 64; h = 2*math.pi/N; x = [h*i for i in xrange(1,N+1)]

# Initial conditions
v = [math.sin(y) for y in x]
alpha = 0.5
t = 0
dt = .001 #Timestep size

# (ik)^2 Vector
I = complex(0,1)
k = numpy.array([I*n for n in range(0,N/2) + [0] + range(-N/2+1,0)])
k2=k**2;

# Setting up Plot
tmax = 5.0; tplot = 0.1
plotgap= int(round(tplot/dt))
nplots = int(round(tmax/tplot))
data = numpy.zeros((nplots+1,N))
data[0,:] = v
tdata = [t]

for i in xrange(nplots):
    v_hat = numpy.fft.fft(v)  # convert to fourier space
    for n in xrange(plotgap):
        v_hat = v_hat / (1-dt*alpha*k2) # backward Euler timestepping

    v = numpy.fft.ifft(v_hat)   # convert back to real space
    data[i+1,:] = numpy.real(v)   # records data

    t = t+plotgap*dt    # records real time
    tdata.append(t)

# Plot using mesh
xx,tt = (numpy.mat(A) for A in (numpy.meshgrid(x,tdata)))

fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(xx, tt, data,rstride=1, cstride=1, cmap=cm.jet,
        linewidth=0, antialiased=False)
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.xlabel('x')
plt.ylabel('t')
plt.show()
```

```python
# %load Heat_Eq_1D_Spectral_FE.py
#!/usr/bin/env python
"""
Solving Heat Equation using pseudo-spectral and Forward Euler
u_t= \alpha*u_xx
BC= u(0)=0, u(2*pi)=0
IC=sin(x)
"""

import math
import numpy
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator

# Grid
N = 64                     # Number of steps
h = 2*math.pi/N                 # step size
x = h*numpy.arange(0,N)    # discretize x-direction

alpha = 0.5                # Thermal Diffusivity constant
t = 0
dt = .001

# Initial conditions
v = numpy.sin(x)
I = complex(0,1)
k = numpy.array([I*y for y in range(0,N/2) + [0] + range(-N/2+1,0)])
k2=k**2;

# Setting up Plot
tmax = 5; tplot = .1;
plotgap = int(round(tplot/dt))
nplots  = int(round(tmax/tplot))

data = numpy.zeros((nplots+1,N))
data[0,:] = v
tdata = [t]

for i in xrange(nplots):
    v_hat = numpy.fft.fft(v)

    for n in xrange(plotgap):
        v_hat = v_hat+dt*alpha*k2*v_hat # FE timestepping

    v = numpy.real(numpy.fft.ifft(v_hat))   # back to real space
    data[i+1,:] = v

    # real time vector
    t = t+plotgap*dt
    tdata.append(t)

# Plot using mesh
xx,tt = (numpy.mat(A) for A in (numpy.meshgrid(x,tdata)))

fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(xx, tt, data,rstride=1, cstride=1, cmap=cm.jet,
        linewidth=0, antialiased=False)
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.xlabel('x')
plt.ylabel('t')
plt.show()
```