# Exercises for EBT617E - Week 2

## Bloch Theorem

An energy eigenstate in a periodic potential can be written as

$$ \Psi_k(z) = \mathrm{e}^{ikz} u_k(z) $$

Since the potential $V(z)$ and the $u_k(z)$ functions are periodic in $z$ with period $a$, they can be written as a Fourier series

$$ V(z) = V(z+a) = \sum_n V_n \mathrm{e}^{i2\pi n z /a} = \sum_n V_n \mathrm{e}^{iG_n z} \qquad \mathrm{and} \qquad  u_k(z) = \sum_n u_{k,n} \mathrm{e}^{iG_n z} $$

where $G_n = \frac{2\pi}{a} n$ are the reciprocal lattice vectors. Since any $k' = (k+G_n)$ with $k \in [-\pi/a,\pi/a)$ in the BZ, we can label the states with the indices $(k,n)$.

The Schrödinger equation

$$ \left[ \frac{\hbar^2 \mathrm{d}^2}{2m \mathrm{d}z^2} + V(z) \right] \Psi_k(z) = E_k \Psi_k(z) $$

can be written in Fourier space as

$$ \epsilon_{k+G_n} u_{k,n}  + \sum_m V_m u_{k,n-m} = E_k u_{k,n} $$

where $\epsilon_{k+G_n} = \frac{\hbar^2 (k+G_n)^2}{2m}$. For example, for a potential with only $V_{\pm 1}$ nonvanishing, we have a tridiagonal band matrix of the following form:

\begin{equation}
    \begin{pmatrix} \ddots &                 &                        &                   &                     &   & & \\
                    \ddots & \ddots          &   \ddots               &                   &                     &   & &  \\
                           & V_{-1}          &  \epsilon_{k,G_{-1}}   &  V_1              &                     &   & &\\
                           &                 &    V_{-1}              &  \epsilon_{k,G_0} &  V_1                &   & &  \\
                           &                 &                        &  V_{-1}           & \epsilon_{k,G_1}    & V_1 & &\\
                           & & & &\ddots &\ddots &\ddots &  \\
                           & & & & & &\ddots &
                                                \end{pmatrix}
                    \begin{pmatrix} \vdots \\ \vdots \\ u_1 \\ u_0 \\ u_{\textit{-}1} \\ \vdots \\ \vdots \end{pmatrix}
                    =
                   E_k \begin{pmatrix} \vdots \\ \vdots \\ u_1 \\ u_0 \\ u_{\textit{-}1} \\ \vdots \\ \vdots \end{pmatrix}
\end{equation}

Scaling:

$$ z' = z / a, \qquad k' = ka, \qquad E' = E / (\hbar^2/2ma^2) $$

so that $k \in [-\pi,\pi)$.

```python

```

```python
import numpy as np

%matplotlib inline
import matplotlib.pyplot as plt
```

```python
# k in [-pi,pi]
def ϵkG(k,n):
    G = 2*np.pi*n
    return (G+k)**2
```

```python
# number of bands is 2*nmax+1
nmax = 50
dB = 2*nmax+1

# number of points in the Brillouin zone
Nk = 2**8

kk, dk = np.linspace(-np.pi,np.pi,Nk,endpoint=True,retstep=True)
```

Potential

$$ V(x) = -V_\mathrm{max} \cos (2 \pi x/a) $$

```python
Vmax = 1

def Vcrystal(x):
    return -Vmax*np.cos(2*np.pi*x)

dF = 1
FCoeffs = np.zeros(2*dF+1)
FCoeffs[0] = -Vmax/2
FCoeffs[1] = 0
FCoeffs[2] = -Vmax/2
```

```python
# real-Fourier space transform

# real space mesh
Nz = 201
zmax = 0.5
zz = np.linspace(-zmax,zmax,Nz,endpoint=True)

nn = np.arange(-dF,dF+1)

expqnz= np.exp(1j*2*np.pi*np.outer(zz,nn))
```

```python
# plot the periodic potential using Fourier coefficients
Vz = expqnz@FCoeffs
plt.plot(zz,Vz.real)
plt.plot(zz,Vz.imag)
plt.xlabel("z/a")
plt.xlabel("V(z)")
```

```python
def FMat(FCoeffs, nmax):
    dB = 2*nmax+1
    MM = np.zeros([dB,dB])
    dM = len(FCoeffs)//2+1
    for iFc,Fc in enumerate(FCoeffs):
        MM+=np.diag([Fc]*(dB-np.abs(-dM+iFc+1)),-dM+iFc+1)
    return MM # +MM.T

def Hband(nmax,k,VFMat):
    nn = np.arange(-nmax,nmax+1)
    HH = np.diag(ϵkG(k,nn))
    HH+= VFMat
    return HH
```

```python
plt.imshow(FMat(FCoeffs, nmax))
```

```python
k=0
VFMat = FMat(FCoeffs, nmax)
Hb    = Hband(nmax,k,VFMat)
```

```python
plt.imshow(Hb)
```

```python
Ebands=[]
for k in kk:
    Hb = Hband(nmax,k,VFMat)
    ee, uu = np.linalg.eigh(Hb)
    Ebands.append(ee)

Ebands=np.array(Ebands)
```

## Density of States

Use broadening of energy levels

$$ \frac{1}{x-x_0-i\epsilon} = \frac{x-x_0}{(x-x_0)^2 + \epsilon^2}+ i\frac{\epsilon}{(x-x_0)^2 + \epsilon^2} $$

$$ \frac{1}{x-x_0-i\epsilon} = \mathcal{P}\frac{1}{(x-x_0)}+ i\pi \delta(x-x_0) $$

Density of states

$$ \rho(E) = \int \mathrm{d}E' \delta(E-E') $$

```python
def get_dos(x,EE,ϵ=0.01):
    return np.imag(np.sum(1/(x-EE-1j*ϵ)))/(len(EE)*np.pi)
```

```python
get_dos(0, Ebands)
```

    1.1797524342104169

```python
Ebands[:,:2].flatten().max()
```

    39.32786474690782

```python
%%time
Earr = np.linspace(0.8*Ebands[:,:2].flatten().min(), Ebands[:,:2].flatten().max(), 401)
ϵ=0.1
Edos = [get_dos(Eval, Ebands[:,:3].flatten(), ϵ=ϵ) for Eval in Earr]
```

    CPU times: user 6.92 ms, sys: 0 ns, total: 6.92 ms
    Wall time: 6.62 ms

```python
fig, ax = plt.subplots(1,3, figsize=(15,5))

Nbands=2
# plot the first two bands
for ni,n in enumerate(nn[:Nbands]):
    ax[0].plot(kk,Ebands[:,ni],'-')
    ρE, Ea = np.histogram(Ebands[:,ni], bins=41, density=True)
    ax[1].plot(ρE,Ea[:-1],'.-')
# energy gap between 1st and 2nd band
# plt.axhline(y=Ebands[-1,0], ls='--', color="gray")
# plt.axhline(y=Ebands[-1,1], ls='--', color="gray")

ax[2].plot(Edos,Earr,'-')

print(f'ΔE = {Ebands[0,1] - Ebands[0,0]:.3}, |2V_G| = {2*abs(FCoeffs[-1]):3}')

#plt.ylim([Ebands[0,0]*.8,Ebands[0,1]*1.2])
ax[0].set_ylabel("E")
ax[0].set_xlabel("ka")
ax[0].set_title("Band Structure")
ax[1].set_xlabel("ρ(E) (arb. units.)")
ax[1].set_title("Density of states");

ax[1].set_ylim([-1,20])
ax[1].axvline(0,ls='--',c='k')
ax[2].set_ylim([-1,20])
ax[2].axvline(0,ls='--',c='k')

```

```python
# real-Fourier space transform
nn = np.arange(-nmax,nmax+1)

# create the transformation matrix for the z-mesh
# for k=0!
expqnz = np.exp(1j*2*np.pi*np.outer(zz,nn))
```

```python
# plot the real space wave function for an eigenstate

for iz in range(3):
    uz = expqnz@uu[:,iz]
    plt.plot(zz,np.abs(uz)**2)
plt.xlabel("z/a")
plt.ylabel("u(z)")
```

```python
Ebands=[]
for k in kk:
    Hb = Hband(nmax,k,VFMat)
    ee, uu = np.linalg.eigh(Hb)
    Ebands.append(ee)

Ebands=np.array(Ebands)
```

# Wannier orbitals

```python
uu.shape
```

    (101, 101)

$$ W_n(z-Z) = \sum_k \psi_{nk}(z) e^{-ikZ} = \sum_k u_{nk}(z) e^{ikz} e^{i\phi_k} e^{-ikZ} $$

```python
Nz = 1001
zmax = 10
zz, dz = np.linspace(-zmax,zmax,Nz,endpoint=True,retstep=True)

# create the transformation matrix for the z-mesh
expqnz= np.exp(1j*2*np.pi*np.outer(zz,nn))

# obtain u_kn(z) functions for each k in BZ
uz_kn = []
for k in kk:
    Hb = Hband(nmax,k,VFMat)
    ee, uu = np.linalg.eigh(Hb)
    uzn = expqnz@uu/Nz
    uz_kn.append(uzn)

uz_kn = np.array(uz_kn)
```

```python
uz_kn.shape
```

    (256, 1001, 101)

```python
# Wannier function position
ZW = 0
expzk = np.exp( np.outer(1j*kk,zz-ZW) )
ZW1 = ZW+1
expzk1 = np.exp( np.outer(1j*kk,zz-ZW1) )
ZW2 = ZW1+1
expzk2 = np.exp( np.outer(1j*kk,zz-ZW2) )
```

```python
# Choose a band index
iband = 0
Wzn = expzk*uz_kn[:,:,iband]
Wzn1 = expzk1*uz_kn[:,:,iband]
Wzn2 = expzk2*uz_kn[:,:,iband]
```

```python
Wzn.shape
```

    (256, 1001)

```python
plt.plot(zz,np.abs(np.sum(Wzn, axis=0))**2,'-')
plt.plot(zz,np.abs(np.sum(Wzn1, axis=0))**2,'-')
plt.plot(zz,np.abs(np.sum(Wzn2, axis=0))**2,'-')

plt.xlabel(r"$z/a$")
plt.ylabel(r"$P(z)$")
```

## Construction of a tight-binding model from Wannier orbitals

Let's just take one orbital from the lowest band.

Overlap integrals

$$ t^{(n)} = \int \mathrm{d}z \, W^*(z-Z_n) \, H(z) \, W(z) $$

Below with second n.n. hoppings the resulting band structure is

$$ E_k = t^{(0)} + 2t^{(1)}\cos(k) + 2t^{(2)}\cos(2k) $$

```python
Wz0 = np.sum(Wzn, axis=0)
Wz1 = np.sum(Wzn1, axis=0)
Wz2 = np.sum(Wzn2, axis=0)
```

```python
# check normalization
NormWz = np.sum(np.conj(Wz0)*Wz0)*dz
NormWz
```

    (0.06437939082421533+0j)

```python
# check orthogonality
np.sum(np.conj(Wz1)*Wz0)*dz
```

    (0.0005164948790171009+4.7898719539267714e-18j)

```python
# onsite energy
Vlat = Vcrystal(zz)
ϵn = np.sum( np.gradient(np.conj(Wz0),dz)*np.gradient(Wz0,dz) )*dz
ϵn += np.sum( np.abs(Wz0)**2*Vlat )*dz

# n.n. hopping
tt = np.sum( np.gradient(np.conj(Wz1),dz)*np.gradient(Wz0,dz) )*dz
tt += np.sum( np.conj(Wz1)*Vlat*Wz0 )*dz

# n.n.n. hopping
tt2 = np.sum( np.gradient(np.conj(Wz2),dz)*np.gradient(Wz0,dz) )*dz
tt2 += np.sum( np.conj(Wz2)*Vlat*Wz0 )*dz
```

```python
ϵn, tt, tt2
```

    ((0.20619156324675686+0j),
     (-0.12351793081370391+8.309866997740776e-18j),
     (0.02649017818354519+2.1409060552784377e-18j))

```python
xf = 1/NormWz
Ntb = Nk

# One can add PBC..
HMtb = np.diag([xf*tt]*(Ntb-1)          , k=1)
HMtb += np.diag([np.conj(xf*tt)]*(Ntb-1), k=-1)

HMtb += np.diag([xf*ϵn.real]*Ntb, k=0)

HMtb += np.diag([xf*tt2]*(Ntb-2)          , k=2)
HMtb += np.diag([np.conj(xf*tt2)]*(Ntb-2), k=-2)
```

```python
plt.imshow(abs(HMtb)[:20,:20])
```

```python
EEtb, UUtb = np.linalg.eigh(HMtb)
```

```python
Neigs = -1
plt.plot(EEtb[:Neigs])
plt.plot(np.sort(Ebands[:,iband])[:Neigs])
```

```python
plt.plot(kk,Ebands[:,0], label="Bloch")
plt.plot(kk, xf.real*(ϵn.real-2*np.abs(tt)*np.cos(kk)+0*np.abs(tt2)*np.cos(2*kk)), label="TB, n.n." )
plt.plot(kk, xf.real*(ϵn.real-2*np.abs(tt)*np.cos(kk)+2*np.abs(tt2)*np.cos(2*kk)), label="TB, n.n.n." )
plt.xlabel("$k$")
plt.ylabel("$E_k$")
plt.legend();
```

```python

```

```python

```

```python

```


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
import warnings
```

# Numerical Quadrature

**Goal:** Accurately Evaluate definite integrals

$$
\int^b_a f(x) dx
$$

using a finite number of samples

### Why?

1. Many integrals do not have closed form solutions

$$
   \int^b_a \sqrt{1 + \cos^2 x} dx
$$

Solution to ordinary differential equations

   $$\frac{\text{d}^2 u}{\text{d}t^2} = f\left(u, \frac{\text{d} u}{\text{d}t}, t \right)$$

   Defining $v = \frac{\text{d} u}{\text{d}t}$ then leads to

   $$\begin{bmatrix}
   \frac{\text{d} v}{\text{d}t} \\ \frac{\text{d} u}{\text{d}t} \end{bmatrix} = \begin{bmatrix} f(u, v, t) \\ v \end{bmatrix}$$

   which can be solved by integration

   $$\begin{bmatrix}
   v \\ u \end{bmatrix} = \begin{bmatrix} v(t_0) + \int^t_{t_0} f(u, v, \hat{t}) d\hat{t} \\ u(t_0) + \int^t_{t_0} v d\hat{t} \end{bmatrix}$$

Solving partial differential equations
$$
    u_t = \nabla^2 u
$$

### Basics of Quadrature

We want to approximate an integral $I$ with some approximation $I_N$ such that
$$
    I = \int^b_a f(x) dx \approx I_N = \sum^{N}_{i=1} w_i f(x_i)
$$
where the $x_i$ are the *quadrature points* or *nodes* and the $w_i$ are the *weights*.  Usually a particular quadrature rule specifies the points $x_i$ resulting in a particular set of weights $w_i$.

Convergence requires that
$$
    \lim_{N \rightarrow \infty} I_N = I.
$$

## Riemann Sums

Given $f(x)$ and a partition of the interval $[a,b]$ with $\{x_i\}^N_{i=0}$ and $a = x_0 < x_1 < \ldots < x_N = b$ and $x^*_i \in [x_i, x_{i+1}]$ we define the Riemann integral as

$$\int^b_a f(x) dx = \lim_{N\rightarrow \infty} \sum^{N-1}_{i=0} f(x_i^*) (x_{i+1} - x_i)$$

This is a general definition and leads to a number of quadrature approaches that depend on how we pick $x_i^* \in [x_i, x_{i+1}]$.

### Midpoint Rule

Choose $x_i^*$ such that

$$x_i^* = \frac{x_{i+1} + x_i}{2}$$

so that

$$I[f] = \int^b_a f(x) dx \approx \sum^{N-1}_{i=0} f\left(\frac{x_{i+1} + x_i}{2} \right ) (x_{i+1} - x_i) = I_N[f]$$

over $\Delta x_i = x_{i+1} - x_i$

#### Example:  Integrate using midpoint rule

Calculate and illustrate the midpoint rule.  Note that we are computing the cummulative integral here:

$$
    \int^x_0 \sin(\hat{x}) d\hat{x} = \left . -\cos \hat{x} \right|^x_0 = 1 - \cos x
$$

```python
# Note that this calculates the cummulative integral from 0.0

f = lambda x: numpy.sin(x)
I = lambda x: 1.0 - numpy.cos(x)
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

num_partitions = 40
x_hat = numpy.linspace(0.0, 2.0 * numpy.pi, num_partitions + 1)
x_star = 0.5 * (x_hat[1:] + x_hat[:-1])
delta_x = x_hat[1] - x_hat[0]
```

```python
fig = plt.figure(figsize=(8,6))
fig.subplots_adjust(hspace=.5)
axes = fig.add_subplot(2, 1, 1)

axes.plot(x, numpy.zeros(x.shape), 'k--')
axes.plot(x, f(x), 'b')

for i in range(num_partitions):
    axes.plot([x_hat[i], x_hat[i]], [0.0, f(x_star[i])], 'k--')
    axes.plot([x_hat[i + 1], x_hat[i + 1]], [0.0, f(x_star[i])], 'k--')
    axes.plot([x_hat[i], x_hat[i + 1]], [f(x_star[i]), f(x_star[i])], 'k--')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Partition and $f(x)$")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-1.1, 1.1))

I_hat = numpy.zeros(x_star.shape)
I_hat[0] = f(x_star[0]) * delta_x
for i in range(1, num_partitions):
    I_hat[i] = I_hat[i - 1] + f(x_star[i]) * delta_x

axes = fig.add_subplot(2, 1, 2)

axes.plot(x, I(x), 'r')
# Offset due to indexing above
axes.plot(x_star + delta_x / 2.0, I_hat, 'ko')

err = numpy.abs(I(x_star) - I_hat)

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Integral and Approximated Integral, Max Err = {}".format(err.max()))
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-0.1, 2.5))

plt.show()
```

### Newton-Cotes Quadrature

Using $N+1$ equally spaced points, evaluate $f(x)$ at these points and exactly integrate the interpolating polynomial:

$$I_N[f] = \int^b_a P_N(x) dx$$

#### Trapezoidal Rule

Use $N = 1$ polynomial to derive the trapezoidal rule.

Trapezoidal rule uses $N = 1$ order polynomials between each point (i.e. piece-wise defined linear polynomials).  Using the Linear Lagrange basis we can write the interpolating polynomial as

$$
    P_1(x) = f(x_i)\frac{x_{i+1}-x}{\Delta x} + f(x_{i+1})\frac{x - x_i}{\Delta x}
$$

where $\Delta x = x_{i+1}-x_i$ is the width of the interval.

Integrating this polynomial we have

\begin{aligned}
    I_N[f] &= \int^{x_{i+1}}_{x_i} f(x_i)\ell_0(x) + f(x_{i+1})\ell_1(x) dx  \\
    &=\frac{1}{\Delta x}  \left. \left[ f(x_i)\left(x_{i+1}x - \frac{x^2}{2}\right) + f(x_{i+1})\left(\frac{x^2}{2}-xx_i\right) \right] \right|_{x_i}^{x_{i+1}} \\
    &= \frac{1}{\Delta x}  \left[ f(x_i)\left(\frac{x_{i+1}^2}{2} -x_ix_{i+1} + x_i^2\right) + f(x_{i+1})\left(\frac{x_{i+1}^2}{2} -x_ix_{i+1} + x_i^2\right) \right]\\
    &= \frac{(x_{i+1} - x_i)^2}{2\Delta x}\left[ f(x_i) + f(x_{i+1})\right]\\
    & = \frac{\Delta x}{2}\left[ f(x_i) + f(x_{i+1}) \right] \\
\end{aligned}

We can also simplify the sum over all the intervals by noting that all but the end points will have total contribution of $\Delta x$ to the entire sum such that

$$
    I_N[f] = \frac{\Delta x}{2} (f(x_0) + f(x_N) ) + \sum^{N-1}_{j=1} \Delta x f(x_j)
$$

This is known as the composite or extended trapezoidal rule.

```python
# Note that this calculates the cummulative integral from 0.0

f = lambda x: numpy.sin(x)
I = lambda x: 1.0 - numpy.cos(x)
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

num_partitions = 20
x_hat = numpy.linspace(0.0, 2.0 * numpy.pi, num_partitions + 1)
delta_x = x_hat[1] - x_hat[0]
```

```python
fig = plt.figure()
fig.subplots_adjust(hspace=.5)
axes = fig.add_subplot(2, 1, 1)

axes.plot(x, numpy.zeros(x.shape), 'k--')
axes.plot(x, f(x), 'b')

for i in range(num_partitions):
    axes.plot([x_hat[i], x_hat[i]], [0.0, f(x_hat[i])], 'k--')
    axes.plot([x_hat[i + 1], x_hat[i + 1]], [0.0, f(x_hat[i+1])], 'k--')
    axes.plot([x_hat[i], x_hat[i + 1]], [f(x_hat[i]), f(x_hat[i+1])], 'k--')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Partition and $f(x)$")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-1.1, 1.1))

I_hat = numpy.zeros(x_hat.shape)
I_hat[0] = (f(x_hat[1]) + f(x_hat[0])) * delta_x / 2.0
for i in range(1, num_partitions):
    I_hat[i] = I_hat[i - 1] + (f(x_hat[i + 1]) + f(x_hat[i])) * delta_x / 2.0

axes = fig.add_subplot(2, 1, 2)

axes.plot(x, I(x), 'r')
# Offset due to indexing above
axes.plot(x_hat + delta_x, I_hat, 'ko')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Integral and Approximated Integral")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-0.1, 2.5))

plt.show()
```

#### Simpson's Rule

Simpson's rule uses $N = 2$ order polynomials between each point (i.e. piece-wise defined quadratic polynomials).

The polynomial has the form

$$P_2(x) = \frac{2 f(x_i)}{\Delta x^2} \left (x - \frac{\Delta x}{2} \right ) (x - \Delta x) - \frac{4 f\left(x_i + \frac{\Delta x}{2}\right)}{\Delta x^2}  x (x - \Delta x) + \frac{2 f(x_{i+1})}{\Delta x^2} x \left (x - \frac{\Delta x}{2} \right )$$

Integrating this polynomial we have

$$
    I_N[f] = \int^{x_{i+1}}_{x_i} P_2(x) dx = \frac{\Delta x}{6} f(x_i) + \frac{2 \Delta x}{3} f\left(x_i + \frac{\Delta x}{2} \right ) + \frac{\Delta x}{6} f(x_{i+1})
$$

We can also show this by using the method of undetermined coefficients.

Use the general form of the quadrature rule and determine weights $w_j$ by using functions we know the solution to.  These functions can be any representation of polynomials up to order $N=2$ however the monomials $1$, $x$, $x^2$ are the easiest in this case.

$$
    I_{\Delta x}[f] = w_0 f(0) + w_1 f(\Delta x / 2) + w_2 f(\Delta x)
$$

$$\begin{aligned}
    &\text{if}~f = 1:  &I[f] =  \int^{\Delta x}_{0} 1 dx = \Delta x  \\
    &\text{if}~f = x:  &I[f] =  \int^{\Delta x}_{0} x dx = \frac{\Delta x^2}{2} \\
    &\text{if}~f = x^2:  &I[f] =  \int^{\Delta x}_{0} x^2 dx = \frac{\Delta x^3}{3}
\end{aligned}$$

What are the corresponding systems of equations?

$$
    I[f] = w_0 f(0) + w_1 f(\Delta x / 2) + w_2 f(\Delta x)
$$

$$\begin{aligned}
    &\text{if}~f = 1:  &I[f] =  \int^{\Delta x}_{0} 1 dx = \Delta x & & I_N[1] &= w_0 + w_1 + w_2 \\
    &\text{if}~f = x:  &I[f] =  \int^{\Delta x}_{0} x dx = \frac{\Delta x^2}{2} & & I_N[x] &= w_1 \frac{\Delta x}{2} + w_2\Delta x\\
    &\text{if}~f = x^2:  &I[f] =  \int^{\Delta x}_{0} x^2 dx = \frac{\Delta x^3}{3} & & I_N[x^2] &= \frac{\Delta x^2}{4} w_1 + w_2\Delta x^2\\
\end{aligned}$$

We then have the system of equations:
$$\begin{aligned}
    w_0 &+& w_1 &+& w_2 &=\Delta x \\
        &\quad& \frac{\Delta x}{2} w_1 &+& \Delta x w_2  &= \frac{\Delta x^2}{2} \\
        &\quad& \frac{\Delta x^2}{4} w_1 &+& \Delta x^2 w_2 &=\frac{\Delta x^3}{6} \\
\end{aligned}$$

or

$$
\begin{bmatrix}
    1 & 1 & 1 \\
    0 & \Delta x / 2 & \Delta x \\
    0 & \Delta x^2 / 4 & \Delta x^2 \\
\end{bmatrix} \begin{bmatrix}
    w_0 \\ w_1 \\ w_2
\end{bmatrix} = \begin{bmatrix}
    \Delta x \\ \Delta x^2 / 2 \\ \Delta x^3 / 3
\end{bmatrix} \Rightarrow \begin{bmatrix}
    1 & 1 & 1 \\
    0 & 1 / 2 & 1 \\
    0 & 1 / 4 & 1 \\
\end{bmatrix} \begin{bmatrix}
    w_0 \\ w_1 \\ w_2
\end{bmatrix} = \begin{bmatrix}
    \Delta x \\ \Delta x / 2 \\ \Delta x / 3
\end{bmatrix}\Rightarrow
\begin{bmatrix}
    1 & 1 & 1 \\
    0 & 1 / 2 & 1 \\
    0 & 0 & -1 \\
\end{bmatrix} \begin{bmatrix}
    w_0 \\ w_1 \\ w_2
\end{bmatrix} = \begin{bmatrix}
    \Delta x \\ \Delta x / 2 \\ -\Delta x / 6
\end{bmatrix}
$$

Leading to

$$ w_2 = \frac{\Delta x}{6} \quad w_1 = \frac{2}{3} \Delta x \quad w_0 = \frac{\Delta x}{6}$$

Another way to write Simpson's rule is to use intervals of three points (similar to one of the ways we did this last time).  The formulation here effectively has a $\Delta x$ half of what the intervals show but is easier to program.

```python
# Note that this calculates the cummulative integral from 0.0

f = lambda x: numpy.sin(x)
I = lambda x: 1.0 - numpy.cos(x)
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

num_partitions = 10
x_hat = numpy.linspace(0.0, 2.0 * numpy.pi, num_partitions + 1)
delta_x = x_hat[1] - x_hat[0]
```

```python
fig = plt.figure()
fig.subplots_adjust(hspace=.5)
axes = fig.add_subplot(2, 1, 1)

axes.plot(x, numpy.zeros(x.shape), 'k--')
axes.plot(x, f(x), 'b')

for i in range(num_partitions):
    axes.plot([x_hat[i], x_hat[i]], [0.0, f(x_hat[i])], 'k--')
    axes.plot([x_hat[i + 1], x_hat[i + 1]], [0.0, f(x_hat[i + 1])], 'k--')
    coeff = numpy.polyfit((x_hat[i], x_hat[i] + delta_x / 2.0, x_hat[i + 1]),
                          (f(x_hat[i]), f(x_hat[i] + delta_x / 2.0), f(x_hat[i+1])), 2)
    x_star = numpy.linspace(x_hat[i], x_hat[i+1], 10)
    axes.plot(x_star, numpy.polyval(coeff, x_star), 'k--')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Partition and $f(x)$")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-1.1, 1.1))

I_hat = numpy.zeros(x_hat.shape)
I_hat[0] = delta_x * (1.0 / 6.0 * (f(x_hat[0]) + f(x_hat[1])) + 2.0 / 3.0 * f(x_hat[0] + delta_x / 2.0))
for i in range(1, num_partitions):
    I_hat[i] = I_hat[i - 1] + delta_x * (1.0 / 6.0 * (f(x_hat[i]) + f(x_hat[i+1])) + 2.0 / 3.0 * f(x_hat[i] + delta_x / 2.0))

axes = fig.add_subplot(2, 1, 2)

axes.plot(x, I(x), 'r')
# Offset due to indexing above
axes.plot(x_hat + delta_x, I_hat, 'ko')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Integral and Approximated Integral")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-0.1, 2.5))

plt.show()
```

## Error Analysis

From before we have a particular quadrature scheme $I_N$ which we can also write as

$$
    I_N[f] = \sum^{N-1}_{i=0} w_i f(x_i).
$$

Define the error $E[f]$ such that

$$I[f] = I_N[f] + E_N[f]$$

The degree of $I_N[f]$ is the integer $N$ such that $E_N[p_i] = 0 \quad \forall i \leq n$ and $\exists p_{n+1}$ such that $E[p_{n+1}] \neq 0$.

### Quadrature Accuracy

We can also use our polynomial analysis from before to analyze the errors made using both of the aforementioned methods.  From Lagrange's theorem we have the remainder term as before which we can use to look at the error

$$R_N(x) = (x - x_0)(x - x_1) \cdots (x- x_N) \frac{f^{(N+1)}(c)}{(N+1)!}$$

and integrate it to find the form and magnitude of the error on a single interval.

To find the total error we must sum the error over all the intervals:

$$I[f] = \sum_{i=0}^N \int^{x_{i+1}}_{x_i} P_N(x) dx + \sum_{i=0}^N \int^{x_{i+1}}_{x_i} R_N(x) dx = I_N[f] + E_N[f]$$

as we defined before.

#### Trapezoidal error

With $N=1$ we have

$$R_1(x) = (x - x_i) (x - x_{i+1}) \frac{f''(c)}{2}$$

Integrating this leads to

$$\int^{x_{i+1}}_{x_i} (x - x_i) (x - x_{i+1}) \frac{f''(c)}{2} dx = \frac{\Delta x^3}{12} f''(c)$$

giving us a form for the error.

If we sum up across all the intervals the total error is

$$E_N[f] = -\frac{\Delta x^3}{12} \sum_{i=0}^{N} f''(c_i)$$

or more illustrative

$$E_N[f] = -\frac{1}{2} \Delta x^2 (b - a) \left [ \frac{1}{N} \sum^{N-1}_{i=0} f''(c_i) \right ]$$

where the expression in the brackets is the mean value of the second derivative over the interval $[a,b]$.  This also shows that the trapezoidal rule converges quadratically as $\Delta x \rightarrow 0$.

#### Simpson's Rule Error

Similarly here we have $N = 2$ and

$$R_2(x) = (x - x_i) \left(x - x_i - \frac{\Delta x}{2} \right) (x - x_{i+1}) \frac{f'''(c)}{3!}$$

Integrating and summing the error contributions we find

$$E_N[f] = -\frac{1}{180} (b - a) \Delta x^4 f^{(4)}(c)$$

Interestingly we have gained two orders of accuracy by increasing the polynomial order by only 1!

### Review:  Newton-Cotes Formulas and errors for a single interval

given $x\in[x_i, x_{i+1}]$ and $\Delta x = x_{i+1} - x_i$, all of the   Newton-Cotes quadrature formula's can be derived by integrating the interpolating polynomial through $N$ equi-spaced points across the interval.  The error term is simply the integral of the Lagrange remainder term $R_N(x)$

* Mid-point: 1-point quadrature rule
$$
    I_M[f] = \Delta x\, f(x_i + \Delta x/2) + O(\Delta x^3f'')
$$
* Trapezoidal Rule: 2-point quadrature rule
$$
    I_T[f] = \Delta x\, \left[\frac{1}{2}f(x_i) +\frac{1}{2}f(x_{i+1})\right] + O(\Delta x^3f'')
$$
* Simpson's Rule: 3-point quadrature rule
$$
    I_S[f] = \Delta x\, \left[ \frac{1}{6}f(x_i) +  \frac{2}{3}f(x_i +\Delta x/2) +\frac{1}{6}f(x_{i+1})\right] + O(\Delta x^5f^{iv})
$$

Note: error term converges as N increases ($\Delta x$ gets smaller)

```python
func = lambda x: 1 - x**2 + numpy.sin(5*x)
x0 = 0.
x1 = .5
x_buf = .2
x = numpy.linspace(x0 - x_buf, x1+ x_buf, 100)
x_interval = numpy.linspace(x0, x1, 100)

x_samp = numpy.array([ 0.5*(x0 + x1) ])
f_samp = func(x_samp)

fig = plt.figure(figsize=(15,5))
axes = fig.add_subplot(1,3,1)
axes.plot(x,func(x),linewidth=3)
axes.plot(x_samp, f_samp, 'ro')
axes.plot(x_interval, f_samp*numpy.ones(x_interval.shape), 'k--')
axes.plot(x0*numpy.ones(2), [0., f_samp],'k--')
axes.plot(x1*numpy.ones(2), [0., f_samp],'k--')
axes.plot(x0*numpy.ones(2), [0., func(x0)],'b')
axes.plot(x1*numpy.ones(2), [0., func(x1)],'b')
axes.plot(x, numpy.zeros(x.shape),'k')
axes.text(x0-.02,-.2,'$x_i$',fontsize=16)
axes.text(x1-.02,-.2,'$x_{i+1}$',fontsize=16)
axes.set_title('Midpoint Rule',fontsize=16)
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.set_ylim((-0.25, 2.))
axes.fill_between(x_interval,0.,func(x_interval))
axes.fill_between(x_interval, 0, f_samp*numpy.ones(x_interval.shape), hatch='/', alpha=0)
axes.grid()

axes = fig.add_subplot(1,3,2)
x_samp = numpy.array([x0, x1])
f_samp = func(x_samp)
axes.plot(x,func(x),linewidth=3)
axes.plot(x_samp, f_samp, 'ro')
axes.plot(x_samp, f_samp, 'k--')
axes.plot(x0*numpy.ones(2), [0., f_samp[0]],'k--')
axes.plot(x1*numpy.ones(2), [0., f_samp[1]],'k--')
axes.plot(x0*numpy.ones(2), [0., func(x0)],'b')
axes.plot(x1*numpy.ones(2), [0., func(x1)],'b')
axes.plot(x, numpy.zeros(x.shape),'k')
axes.text(x0-.02,-.2,'$x_i$',fontsize=16)
axes.text(x1-.02,-.2,'$x_{i+1}$',fontsize=16)
axes.set_title('Trapezoidal Rule',fontsize=16)
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.set_ylim((-0.25, 2.))
axes.fill_between(x_interval,0.,func(x_interval))
axes.fill_between(x_samp,0.,f_samp, hatch='/', alpha=0)

axes.grid()

axes = fig.add_subplot(1,3,3)
x_samp = numpy.array([x0, (x0 + x1)/2., x1])
f_samp = func(x_samp)
p = numpy.polyfit(x_samp, f_samp, 2)
f_p2 = numpy.polyval(p, x_interval)
axes.plot(x,func(x),linewidth=3)
axes.plot(x_samp, f_samp, 'ro')
axes.plot(x_interval, f_p2, 'k--')
axes.plot(x0*numpy.ones(2), [0., f_samp[0]],'k--')
axes.plot(x1*numpy.ones(2), [0., f_samp[-1]],'k--')
axes.plot(x0*numpy.ones(2), [0., func(x0)],'b')
axes.plot(x1*numpy.ones(2), [0., func(x1)],'b')
axes.plot(x, numpy.zeros(x.shape),'k')
axes.text(x0-.02,-.2,'$x_i$',fontsize=16)
axes.text(x1-.02,-.2,'$x_{i+1}$',fontsize=16)
axes.set_title('Simpsons Rule',fontsize=16)
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.set_ylim((-0.25, 2.))
axes.fill_between(x_interval,0.,func(x_interval))
axes.fill_between(x_interval,0.,f_p2, hatch='/', alpha=0)

axes.grid()
plt.show()
```

##### Example 1:

If $f(x) = \sin \pi x$ look at the relative accuracy of midpoint, trapezoidal and simpson's rules for a single interval $x\in[0,1]$.

$$\begin{aligned}
    \text{Exact:}  &I[f] = \int^1_0 \sin \pi x = \left . \frac{-\cos \pi x}{\pi} \right |^1_0 = \frac{2}{\pi} \approx 0.636619772 \\
    \text{Midpoint:}  &I_M[f] = \Delta x f(1/2) = \sin (\pi / 2) = 1 \\
    \text{Trapezoid:}  &I_T[f] = \frac{\Delta x}{2} (\sin(0) + \sin(\pi)) = 0 \\
    \text{Simpson's:}  &I_S[f] = \frac{\Delta x}{6} \sin(0) + \frac{2 \Delta x}{3} \sin(\pi / 2) + \frac{\Delta x}{6} \sin(\pi) = \frac{2 \Delta x}{3} = \frac{2}{3}
\end{aligned}$$

### Cumulative (global) Error for extended Newton-Cotes formulas

For $N$ panels of any given N-C quadrature formula over the interval $x\in[a,b]$, the approximate integral and error formulas are

\begin{align}
I[f] &= \sum_{i=0}^N I_i + E_N\\
\end{align}
where
$$
    E_N = (b-a)O(\Delta x^p f^{m}(c))
$$

* Extended Mid-point:
$$
    E_{N_M} = (b-a) O(\Delta x^2 f''')
$$
* Extended Trapezoidal Rule:
$$
    E_{N_M} = (b-a) O(\Delta x^2 f''')
$$
* Extended Simpson:
$$
    E_{N_M} = (b-a) O(\Delta x^4 f^{(v)})
$$

Where $\Delta x = \frac{b -a }{N}$

```python
# Compute the error as a function of delta_x for each method
f = lambda x: numpy.sin(numpy.pi * x)

num_partitions = numpy.array([ 2**n for n in range(0, 16)])
delta_x = numpy.empty(len(num_partitions))
error_mid = numpy.empty(len(num_partitions))
error_trap = numpy.empty(len(num_partitions))
error_simpson = numpy.empty(len(num_partitions))

I_true = 2.0/numpy.pi

for (j, N) in enumerate(num_partitions):
    x_hat = numpy.linspace(0.0, 1.0, N + 1)
    delta_x[j] = x_hat[1] - x_hat[0]

    # Compute Midpoint
    x_star = 0.5 * (x_hat[1:] + x_hat[:-1])
    I_hat = 0.0
    for i in range(0, N):
        I_hat += f(x_star[i]) * delta_x[j]
    error_mid[j] = numpy.abs(I_hat - I_true)/I_true

    # Compute trapezoid
    I_hat = 0.0
    for i in range(1, N):
        I_hat += (f(x_hat[i + 1]) + f(x_hat[i])) * delta_x[j] / 2.0
    error_trap[j] = numpy.abs(I_hat - I_true)/I_true

    # Compute simpson's
    I_hat = 0.0
    for i in range(0, N):
        I_hat += delta_x[j] * (1.0 / 6.0 * (f(x_hat[i]) + f(x_hat[i+1])) + 2.0 / 3.0 * f(x_hat[i] + delta_x[j] / 2.0))
    error_simpson[j] = numpy.abs(I_hat - I_true)/I_true

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_x, error_mid, 'ro', label="Midpoint")
axes.loglog(delta_x, error_trap, 'bo', label="Trapezoid")
axes.loglog(delta_x, error_simpson, 'go', label="Simpson's")
axes.loglog(delta_x, order_C(delta_x[0], error_trap[0], 2.0) * delta_x**2.0, 'b--', label="2nd Order")
axes.loglog(delta_x, order_C(delta_x[0], error_simpson[0], 4.0) * delta_x**4.0, 'g--', label="4th Order")
axes.legend(loc='best')
axes.set_xlabel('$\Delta x$', fontsize=16)
axes.set_ylabel('Relative Error', fontsize=16)
axes.grid()

plt.show()
```

### Recursive Improvement of Accuracy

Say we ran the trapezoidal rule with step size $2 \Delta x$, we then will have

$$\begin{aligned}
			\int^{x_2}_{x_0} f(x) dx  &= \frac{2 \Delta x}{2} (f_0 + f_2) =  h (f_0 + f_2) \Rightarrow \\
			\int^b_a f(x)dx &\approx I_{2\Delta x}[f] = \sum^{N/2-1}_{j=0} \Delta x (f_{2j} + f_{2j+2}) \\
			&= \Delta x (f_{0} + f_{2})  + \Delta x (f_{2} + f_{4})  + \cdots + \Delta x (f_{N-2} + f_{N}) \\
			&= \Delta x\left ( f_0 + f_N +  2 \sum^{N/2-1}_{j=1} f_{2j} \right )
  \end{aligned}
$$

Now compare the two rules for $\Delta x$ and $2 \Delta x$:

$$\begin{align}I_{\Delta x}[f] &= \frac{\Delta x}{2} \left (f_0 + f_N + 2 \sum^{N-1}_{j=1} f_j \right)\\ I_{2 \Delta x}[f] &= \Delta x \left ( f_0 + f_N +  2 \sum^{N/2-1}_{j=1} f_{2j} \right )\end{align}$$

$$I_{\Delta x}[f] = \frac{1}{2} I_{2\Delta x} + \Delta x(f_1 + f_3 + \cdots + f_{N-1})$$

Here we see we can actually reuse the work we did to calculate $Q_{2 \Delta x}[f]$ to refine the integral.

### Arbitrary Intervals (Affine Transforms)

A lot of quadrature rules are defined on specific intervals, e.g $x\in[0,1]$ or $x\in[-1,1]$.  However any two intervals can always be mapped onto each other through an  *affine transform* or *affine map* which is a __linear  transformation__.

$$
    x = \alpha\xi + \beta
$$
where $\alpha$ is a scaling and $\beta$ is a shift.

Given $\alpha$ and $\beta$,  the inverse map is simply

$$
    \xi = \frac{x - \beta}{\alpha}
$$

### Example:

Map $ x \in [a,b]\rightarrow\xi \in [-1,1]$

```python
fig = plt.figure(figsize=(5,5))
axes = fig.add_subplot(1,1,1)
axes.plot(x,func(x),linewidth=3)
axes.fill_between(x_interval,0.,func(x_interval))
axes.plot(x, numpy.zeros(x.shape),'k')
axes.text(x0-.02,-.2,'$a$',fontsize=16)
axes.text((x0+x1)/2.-.02,-.2,'$x$',fontsize=16)
axes.text(x1-.02,-.2,'$b$',fontsize=16)
axes.set_xlabel('x', fontsize=16)
axes.set_ylabel('f(x)', fontsize=16)
axes.text(x0-.06,-.4,'$-1$',fontsize=16)
axes.text((x0+x1)/2.-.02,-.4,'$\\xi$',fontsize=16)
axes.text(x1-.02,-.4,'$1$',fontsize=16)
axes.set_ylim((-0.5, 2.))
axes.grid()

```

$$x =  \frac{b - a}{2}\xi + \frac{a+b}{2}$$

$$\xi = \left( x - \frac{a + b}{2}\right) \frac{2}{b-a}$$

Given an Affine mapping,  it is easy to transform any integral from one coordinate system to another

$$
    I[f] = \int^b_a f(x) dx = \int^1_{-1} f(x(\xi)) \frac{dx}{d\xi} d\xi = \frac{b - a}{2} \int^1_{-1} f(x(\xi)) d\xi\\
$$

and the $N$ point quadrature rule for a single interval is
$$
    I_N[f]_i = \frac{\Delta x_i}{2} \sum_{n=1}^{N} w_n f(x(\xi_n))
$$

and the cumulative integral over $M$ panels
$$
    I_N[f] = \sum_{i=1}^M I_N[f]_i
$$

#### Example:  Newton-Cotes Rules

We can rewrite our previous quadrature rules so that they are given on the interval $\xi \in [-1, 1]$ instead of $x \in [x_i, x_{i+1}]$.  Recall that a general quadrature rule can be written as
$$
    \sum^N_{i=1} w_i f(\xi_i)
$$
where $w_i$ are the weights and $\xi_i$ are the points specified to evaluate the function at.

 For Newton-Cotes rules we know that the points $\xi_i$ are uniformly distributed on $[-1, 1]$ but we still need to define the weights.  For a 2-Point trapezoid rule we can do this to find that
$$
    \int^1_{-1} f(x) dx \approx f(-1) + f(1)
$$
so the quadrature points are at  $\xi_0 = [-1, 1]$ with weights $w = [1, 1]$

Note that if we map this using our affine transform we would get back the original trapezoid rule:
$$
    I_2[f] = \frac{dx}{d\xi}\sum_{n=1}^{2} w_n f(x(\xi_n))  = \frac{\Delta x}{2}(f(a) + f(b))
$$

Note: $dx = \Delta x$, and $d\xi = 2$ (\[-1, 1\])

Similarly for Simpson's rule we have
$$
    \xi = [-1, 0, 1] \quad \text{and} \quad w = \left[\frac{1}{3}, \frac{4}{3}, \frac{1}{3} \right].
$$

```python
def ncintegration(f, a, b, N):
    """ Approximate \int_a^b f(x)dx using  Newton-Cotes quadrature rules with N<=3 quadrature points"""

    assert (N > 0)
    assert (N < 4)
    # Build up a nested list of integration points and weights for the interval [-1, 1]:
    xNC = []
    wNC = []
    # 1 point Newton-Cotes (Mid-point)
    xNC.append(numpy.array([0.]))
    wNC.append(numpy.array([2.]))
    # 2 point Newton-Cotes quadrature (Trapezoidal)
    xNC.append(numpy.array([ -1., 1. ]))
    wNC.append(numpy.array([ 1. , 1.]))
    # 3 point Newton-Cotes quadrature (Simpson's rule)
    xNC.append(numpy.array([-1., 0., 1.]))
    wNC.append(numpy.array([1., 4., 1.])/3.)

    # make affine map between x and xi
    xi_map = lambda xi: (b - a)/2. * xi + (a + b) / 2.

    # Calculate I[f] for each NC method
    I = (b - a)/2. * numpy.sum(wNC[N-1]*f(xi_map(xNC[N-1])))
    return I

```

```python
# Note that this calculates the cumulative integral from 0.0

f = lambda x: numpy.sin(x)
I = lambda x: - numpy.cos(x)

a = numpy.pi/4.
b = 3*numpy.pi/4.
b = numpy.pi/2
I_true = I(b) - I(a)
print(I_true)
```

    0.7071067811865475

```python
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)
x_interval = numpy.linspace(a, b, 100)
fig = plt.figure(figsize=(5,5))
axes = fig.add_subplot(1,1,1)
axes.plot(x,f(x),linewidth=3)
axes.fill_between(x_interval,0.,f(x_interval))
axes.plot(x, numpy.zeros(x.shape),'k')

axes.grid()
```

```python
# Calculate Newton-Cotes Quadrature and relative errors for 1,2, and 3 point NC quadrature schemes
for N in range(1,4):
    I = ncintegration(f,a,b,N)
    err = numpy.abs(I - I_true)/numpy.abs(I_true)
    print('N = {}, I_N = {}, err = {}'.format(N, I, err))

```

    N = 1, I_N = 0.7256132880348577, err = 0.026172152977030898
    N = 2, I_N = 0.670379265333622, err = 0.051940551031480015
    N = 3, I_N = 0.7072019471344457, err = 0.00013458497419382203

## Optimal Quadrature Methods

Newton-Cotes formulas assume a fixed, regularly spaced set of quadrature points and calculate quadrature rules based on integrating the interpolating polynomial.

Can we do better?

Actually, if we allow both the points and weights to be chosen in an optimal way, we can generate optimal quadrature rules with $N$ points that can exactly integrate polynomials up to order $2N-1$

### Generalized Gaussian Quadrature

Given $g(x) \in P_N(x)$ with roots $\{x_i\}^N_{i=1}$ with the property
$$
    \int^1_{-1} w(x) x^i g(x) dx = 0 \quad \forall i < N,
$$

implies that  $g(x)$ is __*orthogonal* to all monomials__ $x^i$ ($i <N$) with respect to the weight function $w(x)$.

Note: see Wikipedia page on Gaussian Quadrature

Recall something similar:
$$
    <x, y> = \sum^N_{i=1} x_i y_i = ||x|| \cdot ||y|| \cos \theta.
$$
If $<x, y> = 0$ then the vectors $x$ and $y$ are orthogonal.

Given the above $g(x)$ there then exists a set of weights $\{w_i\}$ such that

$$\int^1_{-1} w(x) P_j(x) dx = \sum^N_{i=1} w_i P_j(x_i) \quad \forall j \leq 2 N - 1$$

In other words, given a polynomial basis function and weight and orthogonality to all polynomials of order $i < N$ we can exactly integrate polynomials of order $2 N - 1$.  For a proof and considerably more detail see https://en.wikipedia.org/wiki/Gaussian_quadrature

Choosing the correct weighting function and basis leads to a number of useful quadrature approaches:

#### Gauss-Legendre Quadrature

General Gauss-Legendre quadrature uses $w(x) = 1$ and $g(x) = \ell_N(x)$ which are the Legendre Polynomials, which can be shown to have weights

$$w_i = \frac{2}{(1-x_i^2)(\ell'_n(x_i))^2}$$

and $x_i$ is the $i$th root of $\ell_N$.

Note: $\ell_N(x)$ here denotes Legendre polynomial, not Lagrange

The first few Legendre Polynomials are
\begin{align}
    \ell_1(x) &=x \\
    \ell_2(x) &= \frac{1}{2}(3x^2 -1)\\
    \ell_3(x) &= \frac{1}{2}x(5x^2 - 3)\\
       \vdots
\end{align}

```python
from scipy.special import legendre
x = numpy.linspace(-1,1,100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
for N in range(5):
    Pn = legendre(N)
    axes.plot(x, Pn(x), label='N={}'.format(N))

axes.grid()
axes.set_title('Legendre Polynomials', fontsize=18)
axes.set_xlabel('x',fontsize=16)
axes.legend(loc='best')
plt.show()

```

Which lead to the quadrature points and weights...

$$N$$	 $$x_i$$	  $$w_i$$

$$1$$	            $$0$$ 	  $$2$$

$$2$$	            $$\pm \sqrt{\frac{1}{3}}$$ 	  $$1$$

$$3$$	  $$0$$ 	  $$8/9$$

                      $$\pm \sqrt{\frac{3}{5}}$$ 	  $$5/9$$

$$4$$	  $$\pm \sqrt{\frac{3}{7} - \frac{2}{7} \sqrt{\frac{6}{5}}}$$	  $$\frac{18 + \sqrt{30}}{36}$$

                      $$\pm \sqrt{\frac{3}{7} + \frac{2}{7} \sqrt{\frac{6}{5}}}$$	 $$\frac{18 - \sqrt{30}}{36}$$

$$5$$	  $$0$$ 	  $$\frac{128}{225}$$

                      $$\pm \frac{1}{3} \sqrt{5 - 2 \sqrt{\frac{10}{7}}}$$	  $$\frac{322 + 13\sqrt{70}}{900}$$

                      $$\pm \frac{1}{3} \sqrt{5 + 2 \sqrt{\frac{10}{7}}}$$	  $$\frac{322 - 13\sqrt{70}}{900}$$

Note: weights always add up to 2 for \[-1,1\] for each N.

##### Example 2:  2-Point Gauss-Legendre Quadrature

Let $N=2$ on $x \in [-1,1]$

$$I_2[f] = w_0 f(x_0) + w_1 f(x_1)$$

From above table we are told that $x_0 = -\sqrt{\frac{1}{3}}$, $x_1 = \sqrt{\frac{1}{3}}$ and $w_0 = w_1 = 1$

We can actually solve for the 2 weights and 2 locations using the method of *undetermined coefficients*

I.e.  Assume
$$
    I_2[f] = w_0 f(x_0) + w_1 f(x_1)
$$

and solve for $w_0, w_1, x_0, x_1$ that exactly integrate the first 4 monomials, $1, x, x^2, x^3$.

$$\begin{aligned}
    &\text{if}~f = 1:  &I[f] =  \int^{1}_{-1} 1 dx = 2 & & I_2[1] &= w_0 + w_1\\
    &\text{if}~f = x:  &I[f] =  \int^{1}_{-1} x dx = 0 & & I_2[x] &= w_0 x_0 + w_1 x_1\\
    &\text{if}~f = x^2:  &I[f] =  \int^{1}_{-1} x^2 dx = \frac{2}{3} & & I_2[x^2] &= w_0 x_0^2 + w_1 x_1^2\\
    &\text{if}~f = x^3:  &I[f] =  \int^{1}_{-1} x^3 dx = 0 & & I_2[x^3] &= w_0 x_0^3 + w_1 x_1^3\\
\end{aligned}$$

$$\begin{aligned}
    &w_0 + w_1 = 2\\
    &w_0 x_0 + w_1 x_1 = 0\\
    &w_0 x_0^2 + w_1 x_1^2 = \frac{2}{3}\\
    &w_0 x_0^3 + w_1 x_1^3 = 0\\
\end{aligned}$$

Note that we need to solve for 4 unknowns $x_0$, $x_1$, $w_0$, and $w_1$.  Solving these non-linear equations leads to

$$x_0 = -\sqrt{\frac{1}{3}}, x_1 = \sqrt{\frac{1}{3}}  \quad \text{and} \quad w_0 = w_1 = 1 $$

Note: Integration is linear transformation

Note: Quadrature points for GL are never at endpoints of interval

##### Example: Gauss-Legendre Quadrature single interval

let $f(x) = \cos(x)$:

$$
    I[f] = \int_0^{\pi/2} f(x) dx = \left.\sin(x)\right|_0^{\pi/2} = 1.
$$

so the affine map from $x\in[0,\pi/2]\rightarrow\xi\in[-1,1]$ is

\begin{align}
x &= \frac{\pi}{4}\xi + \frac{\pi}{4} \\
&= \frac{\pi}{4}\left(\xi + 1\right)
\end{align}

```python
f = lambda x: numpy.cos(x)
I = 1.

x = numpy.linspace(-1.,2, 100)
x_interval = numpy.linspace(0., numpy.pi/2, 100)
fig = plt.figure(figsize=(5,5))
axes = fig.add_subplot(1,1,1)
axes.plot(x,f(x),linewidth=3)
axes.fill_between(x_interval,0.,f(x_interval))
axes.plot(x, numpy.zeros(x.shape),'k')

axes.grid()
```

```python
def glintegration(f, a, b, N):
    """ Approximate \int_a^b f(x)dx using  Gauss-Legendre quadrature rules with N<=4 quadrature points"""

    # Build up a nested list of gauss points and weights:
    xGl = []
    wGl = []
    # 1 point Gauss-legendre quadrature
    xGl.append(numpy.array([0.]))
    wGl.append(numpy.array([2.]))
    # 2 point Gauss-legendre quadrature
    xGl.append(numpy.array([ -1./numpy.sqrt(3.), 1./numpy.sqrt(3)]))
    wGl.append(numpy.array([ 1. , 1.]))
    # 3 point Gauss-legendre quadrature
    xGl.append(numpy.array([-numpy.sqrt(3. /5.), 0., numpy.sqrt(3. / 5.)]))
    wGl.append(numpy.array([5., 8., 5.])/9.)
    # 4 point Gauss-Legendre quadrature
    xGl.append(numpy.array([-numpy.sqrt(3.0 / 7.0 - 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
           numpy.sqrt(3.0 / 7.0 - 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
          -numpy.sqrt(3.0 / 7.0 + 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
           numpy.sqrt(3.0 / 7.0 + 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0))]))
    wGl.append(numpy.array([(18.0 + numpy.sqrt(30.0)) / 36.0, (18.0 + numpy.sqrt(30.0)) / 36.0,
         (18.0 - numpy.sqrt(30.0)) / 36.0, (18.0 - numpy.sqrt(30.0)) / 36.0]))

    # make affine map between x and xi
    xi_map = lambda xi: (b - a)/2. * xi + (a + b) / 2.

    I = (b - a)/2. * numpy.sum(wGl[N-1]*f(xi_map(xGl[N-1])))
    return I

```

```python
a = 0.
b = numpy.pi/2
I_true = 1.
err = numpy.zeros(4)
for N in range(1,5):
    I = glintegration(f, a, b, N)
    err[N-1] = numpy.abs(I_true - I)
    print('N = {}, I = {}, err = {}'.format(N,I, err[N-1]))
```

    N = 1, I = 1.1107207345395915, err = 0.11072073453959153
    N = 2, I = 0.9984726134041148, err = 0.0015273865958852495
    N = 3, I = 1.0000081215554983, err = 8.121555498341593e-06
    N = 4, I = 0.9999999771971155, err = 2.2802884469541596e-08

```python
errNC = numpy.zeros(4)
for N in range(1,4):
    I = ncintegration(f, a, b, N)
    errNC[N-1] = numpy.abs(I_true - I)
    print('N = {}, I = {}, err = {}'.format(N,I, errNC[N-1]))

fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
axes.semilogy(range(1,5),errNC,'bs', markersize=10, label='Newton-Cotes')
axes.semilogy(range(1,5),err,'ro', markersize=10, label='Gauss-Legendre')
axes.set_ylabel('Error')
axes.set_xlabel('quadrature points (N)')
axes.legend(loc='best')
axes.grid()
plt.show()
```

```python
# Cubic f(x): Exact interpolation at/after 2N-1 (N=2)
f = lambda x: - 4*x**3 - 3.*x**2 + 2.*x +300.
int_f = lambda x:  - x**4 - x**3. + x**2 + x*300.

# 4th order f(x): N=3
f = lambda x: x**5 - 4*x**3 - 3.*x**2 + 2.*x +300.
int_f = lambda x: x**6/6. - x**4 - x**3. + x**2 + x*300.

a = 1.
b = 4.
I_true = int_f(b) - int_f(a)
print('I_true = {}'.format(I_true))
for N in range(1,5):
    I = glintegration(f, a, b, N)
    print('N = {}, I = {}'.format(N,I))
```

    I_true = 1279.4999999999998
    N = 1, I = 964.21875
    N = 2, I = 1262.625
    N = 3, I = 1279.5
    N = 4, I = 1279.5000000000002

```python
x = numpy.linspace(a - 1, b + 1, 100)
x_interval = numpy.linspace(a, b, 100)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(x,f(x),linewidth=3)
axes.fill_between(x_interval,0.,f(x_interval))
axes.plot(x, numpy.zeros(x.shape),'k')

axes.grid()
plt.show()
```

```python
# Compute Gauss-Legendre based quadrature and affine transforms
f = lambda x: numpy.sin(x)
I = lambda x: 1.0 - numpy.cos(x)
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

num_partitions = 10
x_hat = numpy.linspace(0.0, 2.0 * numpy.pi, num_partitions + 1)
delta_x = x_hat[1] - x_hat[0]

xi_map = lambda a,b,xi : (b - a) / 2.0 * xi + (a + b) / 2.0
xi_0 = -numpy.sqrt(1.0 / 3.0)
xi_1 =  numpy.sqrt(1.0 / 3.0)
I_hat = numpy.zeros(x_hat.shape)
I_hat[0] = (f(xi_map(x_hat[0], x_hat[1], xi_0)) + f(xi_map(x_hat[0], x_hat[1], xi_1))) * delta_x / 2.0
for i in range(1, num_partitions):
    I_hat[i] = I_hat[i - 1] + (f(xi_map(x_hat[i], x_hat[i+1], xi_0)) + f(xi_map(x_hat[i], x_hat[i+1], xi_1))) * delta_x / 2.0

error = numpy.abs(I_hat - I(x_hat + delta_x))
error[:-1]
```

    array([6.97744637e-06, 2.52446381e-05, 4.78241289e-05, 6.60913206e-05,
           7.30687670e-05, 6.60913206e-05, 4.78241289e-05, 2.52446381e-05,
           6.97744637e-06, 1.66533454e-16])

```python

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, I(x), 'r')
# Offset due to indexing above
axes.plot(x_hat + delta_x, I_hat, 'ko')

axes.set_xlabel("x")
axes.set_ylabel("$f(x)$")
axes.set_title("Integral and Approximated Integral")
axes.set_xlim((0.0, 2.0 * numpy.pi))
axes.set_ylim((-0.1, 2.5))
axes.grid()
plt.show()
```

##### Example 3:

If $f(x) = e^x$ look at the relative accuracy of midpoint, trapezoidal, Simpson's and 2-point Gauss-Legendre quadrature for a single interval $x \in [-1,1]$.

$$\begin{aligned}
    \text{Exact:} &I[f] = \int^1_{-1} e^x = \left . e^x \right |^1_{-1} = e - \frac{1}{e} \approx 2.350402387 \\
    \text{Midpoint:} &I_2[f] = 2 e^0 = 2 \\
    \text{Trapezoid:} &I_2[f] = \frac{2}{2} (e^{-1} + e^1) = e + \frac{1}{e} = 3.08616127 \\
    \text{Simpson's:} &I_2[f] = \frac{2}{6} e^{-1} + \frac{4}{3} e^0 + \frac{2}{6} e^1 = \frac{4}{3} + \frac{1}{3} (e^{-1} + e^1) \approx 2.362053757 \\
    \text{Gauss-Legendre:} &I_2[f] = e^{-\sqrt{\frac{1}{3}}} + e^{\sqrt{\frac{1}{3}}} \approx 2.342696088
\end{aligned}$$

```python
# Compute the error as a function of delta_x for each method
f = lambda x: numpy.sin(numpy.pi * x)
I = 2.0 / numpy.pi

# num_partitions = range(50, 1000, 50)
num_partitions = range(5, 50, 5)
delta_x = numpy.empty(len(num_partitions))
error_trap = numpy.empty(len(num_partitions))
error_simpson = numpy.empty(len(num_partitions))
error_2 = numpy.empty(len(num_partitions))
error_3 = numpy.empty(len(num_partitions))
error_4 = numpy.empty(len(num_partitions))

for (j, N) in enumerate(num_partitions):
    x_hat = numpy.linspace(0.0, 1.0, N)
    delta_x[j] = x_hat[1] - x_hat[0]

    # Compute trapezoid
    I_hat = 0.0
    for i in range(0, N - 1):
        I_hat += (f(x_hat[i + 1]) + f(x_hat[i])) * delta_x[j] / 2.0
    error_trap[j] = numpy.abs(I_hat - I)

    # Compute simpson's
    I_hat = 0.0
    for i in range(0, N - 1):
        I_hat += delta_x[j] * (1.0 / 6.0 * (f(x_hat[i]) + f(x_hat[i+1])) + 2.0 / 3.0 * f(x_hat[i] + delta_x[j] / 2.0))
    error_simpson[j] = numpy.abs(I_hat - I)

    # Compute Gauss-Legendre 2-point
    xi_map = lambda a,b,xi : (b - a) / 2.0 * xi + (a + b) / 2.0
    xi = [-numpy.sqrt(1.0 / 3.0), numpy.sqrt(1.0 / 3.0)]
    w = [1.0, 1.0]
    I_hat = 0.0
    for i in range(0, N - 1):
        for k in range(len(xi)):
            I_hat += f(xi_map(x_hat[i], x_hat[i+1], xi[k])) * w[k]
    I_hat *= delta_x[j] / 2.0
    error_2[j] = numpy.abs(I_hat - I)

    # Compute Gauss-Legendre 3-point
    xi_map = lambda a,b,xi : (b - a) / 2.0 * xi + (a + b) / 2.0
    xi = [-numpy.sqrt(3.0 / 5.0), 0.0, numpy.sqrt(3.0 / 5.0)]
    w = [5.0 / 9.0, 8.0 / 9.0, 5.0 / 9.0]
    I_hat = 0.0
    for i in range(0, N - 1):
        for k in range(len(xi)):
            I_hat += f(xi_map(x_hat[i], x_hat[i+1], xi[k])) * w[k]
    I_hat *= delta_x[j] / 2.0
    error_3[j] = numpy.abs(I_hat - I)

    # Compute Gauss-Legendre 4-point
    xi_map = lambda a,b,xi : (b - a) / 2.0 * xi + (a + b) / 2.0
    xi = [-numpy.sqrt(3.0 / 7.0 - 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
           numpy.sqrt(3.0 / 7.0 - 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
          -numpy.sqrt(3.0 / 7.0 + 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0)),
           numpy.sqrt(3.0 / 7.0 + 2.0 / 7.0 * numpy.sqrt(6.0 / 5.0))]
    w = [(18.0 + numpy.sqrt(30.0)) / 36.0, (18.0 + numpy.sqrt(30.0)) / 36.0,
         (18.0 - numpy.sqrt(30.0)) / 36.0, (18.0 - numpy.sqrt(30.0)) / 36.0]
    I_hat = 0.0
    for i in range(0, N - 1):
        for k in range(len(xi)):
            I_hat += f(xi_map(x_hat[i], x_hat[i+1], xi[k])) * w[k]
    I_hat *= delta_x[j] / 2.0
    error_4[j] = numpy.abs(I_hat - I)

fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)

# axes.plot(delta_x, error)
axes.loglog(delta_x, error_trap, 'o', label="Trapezoid")
axes.loglog(delta_x, error_simpson, 'o', label="Simpson's")
axes.loglog(delta_x, error_2, 'o', label="G-L 2-point")
axes.loglog(delta_x, error_3, 'o', label="G-L 3-point")
axes.loglog(delta_x, error_4, 'o', label="G-L 4-point")

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_x, order_C(delta_x[0], error_trap[0], 2.0) * delta_x**2.0, 'r--', label="2nd Order")
axes.loglog(delta_x, order_C(delta_x[0], error_simpson[0], 4.0) * delta_x**4.0, 'g--', label="4th Order")
axes.loglog(delta_x, order_C(delta_x[1], error_3[1], 5) * delta_x**5, 'b--', label="5th Order")
axes.loglog(delta_x, order_C(delta_x[1], error_4[1], 7.0) * delta_x**7.0, 'g--', label="7th Order")

axes.legend(loc='best',fontsize=14)
axes.set_xlabel('$\Delta x$', fontsize=16)
axes.set_xlim((5e-3, delta_x[0]))
axes.grid()

plt.show()
```

#### Other Quadrature Families

 - Clenshaw-Curtis (Gauss-Chebyshev): If $w(x) = \frac{1}{\sqrt{1 - x^2}}$ and $g(x)$ are Chebyshev polynomials then we know the roots of the polynomials to be $x_i = \cos\left(\frac{2i-1}{2N} \pi \right)$ (the Chebyshev nodes) and we can derive that $w_i = \frac{\pi}{N}$.
 - Gauss-Hermite:  If $w(x) = e^{-x^2}$ and $g(x)$ are Hermite polynomials $H_i(x)$ then
   $$w_i = \frac{2^{N-1} N! \sqrt{\pi}}{N^2 (H_{N-1}(x_i))^2}$$

## SciPy Integration Routines

SciPy has a number of integration routines that we have derived here including general purpose integrators that can control error.

```python
import scipy.integrate as integrate
# integrate?
```


```python
# Preload packages
from pyspark.sql import SparkSession
from pyspark.ml import feature, regression, evaluation, Pipeline
from pyspark.sql import functions as fn, Row
import matplotlib.pyplot as plt
spark = SparkSession.builder.getOrCreate()
sc = spark.sparkContext
# dataframe functions
from pyspark.sql import functions as fn
```

# Random forest for predicting credit score

# A problem with Logistic Regression

While logistic regression is a simple, fast, and effective method. However, the predictions depend linearly on the features. This is, after we have learned the parameters $\theta_0,\theta_1,\dots,\theta_m$, given the features $X$, we have a prediction for the a new point $y$:

\begin{equation}
p(y \mid X) = \frac{1}{1+\exp(-(\theta_0 + \sum_{j>0} x_j \theta_j))}
\end{equation}

Let say we choose to predict that $y$ is 1 if $p(y \mid X) > \frac{1}{2}$, therefore, the decision boundary is given by:

\begin{equation}
\frac{1}{1+\exp(-(\theta_0 + \sum_{j>0} x_j \theta_j))} > \frac{1}{2}
\end{equation}

\begin{equation}
1 > \frac{1+\exp(-(\theta_0 + \sum_{j>0} x_j \theta_j))}{2}
\end{equation}

\begin{equation}
2 > 1+\exp(-(\theta_0 + \sum_{j>0} x_j \theta_j))
\end{equation}

\begin{equation}
1 > \exp(-(\theta_0 + \sum_{j>0} x_j \theta_j))
\end{equation}

By taking the log:

\begin{equation}
0 > -(\theta_0 + \sum_{j>0} x_j \theta_j)
\end{equation}

therefore:

\begin{equation}
0 < \theta_0 + \sum_{j>0} x_j \theta_j
\end{equation}

The decision boundary is exactly when the linear combination $X^T\theta = 0$

## The XOR function

A problem of classic importance that clearly shows when linear boundaries do not work is the XOR function. The [XOR function](https://en.wikipedia.org/wiki/Exclusive_or) stands for eXclusive or and it means "one or the other but not both".

Let's define the function now:

```python
import pandas as pd
import numpy as np
```

```python
X = np.array([
        [0, 0],
        [0, 1],
        [1, 0],
        [1, 1]])
y = np.array([0, 1, 1, 0])
```

```python
pd.DataFrame(np.hstack((X, y.reshape(-1, 1))), columns=['x1', 'x2', 'y'])
```

      x1
      x2
      y

      0
      0
      0
      0

      1
      0
      1
      1

      2
      1
      0
      1

      3
      1
      1
      0

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
```

```python
plt.scatter(X[y==0, 0], X[y==0, 1], c='red');
plt.scatter(X[y==1, 0], X[y==1, 1], c='blue');
plt.xlabel('x1');
plt.ylabel('x2');
plt.legend(['0', '1'], loc='center');
```

The problem with this function is that there is no linear function that correctly can classify the data. It is non-separable.

## Other problems

Sometimes we want to __combine__ features - make them __interact__. But we can't really test all possible combination of features. The space of possible combinations is too large.

## Decision tree

Decision tree recursively partitions the feature space. The decisions are typically simple. Because of the recursive partitioning, the decision can be represented as a tree.

These techniques were originally created to display decisions in a human-redeable fashion. They are therefore easy to read and interpret.

For example, for the XOR problem, is very easy to create a decision tree that solves it with 100% accuracy.

```python
def xor_decision_tree(x):
    x1, x2 = x
    if x1 == 0:
        if x2 == 0:
            return 0
        elif x2 == 1:
            return 1
    elif x1 == 1:
        if x2 == 0:
            return 1
        elif x2 == 1:
            return 0
```

```python
# accuracy
(list(map(xor_decision_tree, X)) == y).mean()
```

    1.0

### Types of decision trees

In mathematical terms, decision tress partition the feature space into, say, $M$ _regions_ $R_1, ..., R_M$. For each region, we build a prediction based on the training datapoints that fall into that region.

#### Regression tree
For a regression problem and a region $m$, for example, we may predict that the region value is the mean of the points that belong to that region, denoted by $\hat{y}_{R_m}$. In this simple approach, we evaluate the performance of a regression tree by the squared error of the regression predictions

\begin{equation}
\sum_{m=1}^M \sum_{i\in R_m} (y_i - \hat{y}_{R_m})^2
\end{equation}

#### Classification tree

A similar approach is taken for classification trees, where the dependent variable $y$ is categorical. The classification trees predicts a class $k$ for region $m$ as the _most common training class_ occuring in that region.

To measure the performance of a classification tree, we often measure the proportion of training examples from class $k$ in region $m$, denoted as $\hat{p}_{mk}$. For a particular region, one performance measure is the Gini index

\begin{equation}
G_m = \sum_{k=1}^K \hat{p}_{mk} (1 - \hat{p}_{mk})
\end{equation}

Intuitively, the index is small when the values $\hat{p}_{mk}$ are close to zero or one.

Another measure of performance is cross entropy

\begin{equation}
D_m = - \sum_{k=1}^K \hat{p}_{mk} \text{log}(\hat{p}_{mk})
\end{equation}

where similarly distributions that are highly peaked have the lowest cross entropy.

At the end, we want to minimize either $\sum_m G_M$ or $\sum_m D_m$.

### Building a decision tree

The search of all possible recursive partitions is intractable. So typically the process is split into two steps.

1. Building. The tree is build from the top down (from one feature to all features) in a greedy fashion until some criteria is met.

    1. We start from a region that has every point
    1. We select a feature and a cutting point that splits a region so that the loss is minimized.
    1. We repeat recursively until a criteria, such as number of examples in a leaf, is met.

1. Pruning or regularization. Given a regularization parameter $\alpha$ and starting from the most complex tree found before, we find a new tree that optimizes the following loss function

\begin{equation}
\sum_{m=1}^M \sum_{x_i \in R_m} L(y_i, \hat{f}(x_i)) + \alpha M
\end{equation}

where $L$ is the loss function either for a regression or classification problem. Since $M$ is the number regions, then $\alpha \mid M \mid$ acts as an $L1$ regularization.

# Problems with decision trees

In one word: __Overfitting__

Question: how many ways can we build a tree for the same dataset?

(from ISRL)

1. The good
    1. Trees are very easy to explain to people. In fact, they are even easier to explain than linear regression!
    1. Some people believe that decision trees more closely mirror human decision-making than do the regression and classification approaches seen in previous chapters.
    1. Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small).
    1. Trees can easily handle qualitative predictors without the need to create dummy variables.

1. The bad
    1. Unfortunately, trees generally do not have the same level of predictive accuracy as some of the other regression and classification approaches seen in this book.
    1. Additionally, trees can be very non-robust. In other words, a small change in the data can cause a large change in the final estimated tree

# Bagging

The problem with decision trees and other learning methods is that they have _high variance_. If fit into two training set, they can produce very different fits.

One solution to this problem is to _average_ the opinions of several models fit to different training sets. Given that we usually don't have many training sets, we use a technique called _bootstrap_ to produce them. In boostrap, we generate _fictitious_ training sets by _sampling with replacement_ the training set. We do this multiple times, fitting a model to each bootsrap sample, and producing a predicting based on the average of the trees. This is called __bagging__.

## The wisdom of the crowds

One easy way to understand why bagging helps. Image asking $B$ people the opinion about the height of someone. Assume that each person has a high variance $\sigma^2$ in their estimates and no biases. If we average their opinions, what is the variance of the resulting estimate?

```python
import numpy as np
```

```python
person_std = np.sqrt(10.)
n_simulations = 10000
B = 10
opinions = np.random.normal(scale=person_std, size = (n_simulations, B))
```

```python
# lets average opinions
avg_opinions = opinions.mean(axis=1)
```

```python
# let's look at the variance
np.var(avg_opinions)
```

    0.9806098202243334

```python
import matplotlib.pyplot as plt
```

```python
plt.clf();
plt.plot(np.linspace(1, 10, num=10), 10/np.linspace(1, 10, num=10));
plt.xlabel('B');
plt.ylabel('Variance');
plt.ylim([0, 12]);
```

The variance seems to be a 10th of the original variance! Which is exacty right.

## Tree bagging

Similarly to the wisdom of the crowds, you can think of the statistical estimation of models and then bagging as a way of lowering the variance of models!

Therefore, if we fit ($\hat{f}^{*b}$) different decision tree to boostrapped version $b$ of the data, then we can average those results for regression:

\begin{equation}
\hat{f}_{\text{bagging}}(X) = \frac{1}{B} \sum_{b=1}^B  \hat{f}^{*b}(X)
\end{equation}

or in classification

\begin{equation}
\hat{f}_{\text{bagging}}(X) = \text{majority} \{\hat{f}^{*b}(X) \mid b \in \{1, \dots, B \} \}
\end{equation}

The problem with tree bagging is that we have lost the interpretability of results because we are now averaging the prediction of several trees.

One solution to this is to compute, across all trees, how much on average a feature decreases the loss function. We can then use that measure to sort features by importance.

One convenient by-product of bagging is that we can compute the validaton performance of our model by computing the performance of the fit in the _out of bag_ datapoints. This is, every time we sample with replacement from the data, there is a set of datapoints that is not used in the data.

The argument goes like this. A single point has a chance of $1/n$ of being sampled. This means that the probability of a point not being sampled during a bootstrap sample is $(1-\frac{1}{n})^n$. When $\lim_{n\rightarrow \infty} (1-\frac{1}{n})^n = \frac{1}{e} \approx 36% \approx 1/3 $. So, in boostrapped estimators a point will be out of the fitting 1/3 of the time.

At the end, we will have $B/3$ out of sample estimators for each data which gives us testing performance estimation for free!

### Limitations of the wisdom of the crowds

Now, the wisdom of the crowds assumes that _opinions are independent_, but boostrapping makes the trees dependent because we are resampling the same training data multiple times.

What is the effect of making opinions dependent?

```python
# wisdom of the crowds with opinion correlation
corr = 0.9
B = 1000
cov = (np.ones((B, B)) * corr + np.eye(B)*(1-corr))*(person_std**2)
mu = np.zeros(B)
correlated_opinions = np.random.multivariate_normal(mu, cov, (n_simulations,))
```

```python
correlated_opinions.mean(1).var()
```

    8.908816178750739

With 90% correlation, the crowd doesn't do much better than individuals! In general, the variance of correlated opinions is:

\begin{equation}
\text{var}(f) = \bigl( \frac{1-\rho}{B} + \rho \bigr) \sigma^2
\end{equation}

```python
plt.clf();
pl1 = plt.plot(np.linspace(1, 10, num=10), 10/np.linspace(1, 10, num=10));
pl2 = plt.plot(np.linspace(1, 10, num=10), 10*((1-0.9)/np.linspace(1, 10, num=10) + 0.9));
plt.xlabel('B');
plt.ylabel('Variance');
plt.ylim([0, 12]);
plt.legend(['no correlation', '90% correlation']);
```

## Random forest

Random forest is a tweak to the boostrapped trees predictions that decorrelate trees. It does this by randomly sampling $m$ features during each split when bulding the tree. In this way, each of trees look at different features and become decorrelated. A typical value of $m = \sqrt{p}$, where $p$ is the total number of variables.

## Gradient boosting

Boosting _grows_ a model. It first fits a simple model and the next model focuses on samples poorly fit by the previous model. It is a way of _adaptively_ fitting the data. In a sense, boosting does not resample the data.

Boosting is a technique whose theoretical workings go outside of the scope of this course. Roughly speaking, however, while bagging aims at lowering the variance in the learning process, boosting aims at reducing the bias.

# Application in Spark

```python
# dataframe functions
from pyspark.sql import functions as fn
```

```python
import os
```

```python
credit_score_df = spark.read.parquet('datasets/cs-training.parquet')
# change type to double, drop NAs, and change dependent variable name to label
credit_score_df = credit_score_df.withColumnRenamed('SeriousDlqin2yrs', 'label')
```

```python
credit_score_df.limit(10).toPandas()
```

      case_id
      label
      RevolvingUtilizationOfUnsecuredLines
      age
      NumberOfTime30-59DaysPastDueNotWorse
      DebtRatio
      MonthlyIncome
      NumberOfOpenCreditLinesAndLoans
      NumberOfTimes90DaysLate
      NumberRealEstateLoansOrLines
      NumberOfTime60-89DaysPastDueNotWorse
      NumberOfDependents

      0
      1.0
      1.0
      0.766127
      45.0
      2.0
      0.802982
      9120.0
      13.0
      0.0
      6.0
      0.0
      2.0

      1
      2.0
      0.0
      0.957151
      40.0
      0.0
      0.121876
      2600.0
      4.0
      0.0
      0.0
      0.0
      1.0

      2
      3.0
      0.0
      0.658180
      38.0
      1.0
      0.085113
      3042.0
      2.0
      1.0
      0.0
      0.0
      0.0

      3
      4.0
      0.0
      0.233810
      30.0
      0.0
      0.036050
      3300.0
      5.0
      0.0
      0.0
      0.0
      0.0

      4
      5.0
      0.0
      0.907239
      49.0
      1.0
      0.024926
      63588.0
      7.0
      0.0
      1.0
      0.0
      0.0

      5
      6.0
      0.0
      0.213179
      74.0
      0.0
      0.375607
      3500.0
      3.0
      0.0
      1.0
      0.0
      1.0

      6
      8.0
      0.0
      0.754464
      39.0
      0.0
      0.209940
      3500.0
      8.0
      0.0
      0.0
      0.0
      0.0

      7
      10.0
      0.0
      0.189169
      57.0
      0.0
      0.606291
      23684.0
      9.0
      0.0
      4.0
      0.0
      2.0

      8
      11.0
      0.0
      0.644226
      30.0
      0.0
      0.309476
      2500.0
      5.0
      0.0
      0.0
      0.0
      0.0

      9
      12.0
      0.0
      0.018798
      51.0
      0.0
      0.531529
      6501.0
      7.0
      0.0
      2.0
      0.0
      2.0

| Variable Name                        | Description                                                                                | Type       |
|--------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
| SeriousDlqin2yrs                     | Person experienced 90 days past due delinquency or worse                                                                                | Y/N        |
| RevolvingUtilizationOfUnsecuredLines | Total balance on credit cards and personal lines of credit except real estate and no installment debt like car loans divided by the sum of credit limits | percentage |
| age                                  | Age of borrower in years                                                                                | integer    |
| NumberOfTime30-59DaysPastDueNotWorse | Number of times borrower has been 30-59 days past due but no worse in the last 2 years.                                                                  | integer    |
| DebtRatio                            | Monthly debt payments, alimony,living costs divided by monthy gross income                                                                               | percentage |
| MonthlyIncome                        | Monthly income                                                                                | real       |
| NumberOfOpenCreditLinesAndLoans      | Number of Open loans (installment like car loan or mortgage) and Lines of credit (e.g. credit cards)                                                     | integer    |
| NumberOfTimes90DaysLate              | Number of times borrower has been 90 days or more past due.                                                                                | integer    |
| NumberRealEstateLoansOrLines         | Number of mortgage and real estate loans including home equity lines of credit                                                                           | integer    |
| NumberOfTime60-89DaysPastDueNotWorse | Number of times borrower has been 60-89 days past due but no worse in the last 2 years.                                                                  | integer    |
| NumberOfDependents                   | Number of dependents in family excluding themselves (spouse, children etc.)                                                                              | integer    |

We need to put the independent variables into one feature vector

```python
from pyspark.ml.feature import VectorAssembler
```

```python
training_df, validation_df, testing_df = credit_score_df.randomSplit([0.6, 0.3, 0.1])
```

```python
# build a pipeline for analysis
va = VectorAssembler().setInputCols(training_df.columns[2:]).setOutputCol('features')
```

```python
from pyspark.ml.classification import LogisticRegression, RandomForestClassifier
```

```python
from pyspark.ml import Pipeline
```

```python
lr = LogisticRegression(regParam=0.1)
```

```python
lr_pipeline = Pipeline(stages=[va, lr]).fit(training_df)
```

```python
rf = RandomForestClassifier()
```

```python
rf_pipeline = Pipeline(stages=[va, rf]).fit(training_df)
```

```python
from pyspark.ml.evaluation import BinaryClassificationEvaluator
```

```python
bce = BinaryClassificationEvaluator()
```

```python
bce.evaluate(lr_pipeline.transform(validation_df))
```

    0.6692505322633585

```python
bce.evaluate(rf_pipeline.transform(validation_df))
```

    0.8402624304893843

Without much changes, Random Forest performs much better!

# Try to interpret the results

## Logistic regression

```python
lr_model = lr_pipeline.stages[-1]
```

```python
import pandas as pd
```

```python
pd.DataFrame(list(zip(credit_score_df.columns[2:], lr_model.coefficients.toArray())),
            columns = ['column', 'weight']).sort_values('weight')
```

      column
      weight

      1
      age
      -0.009877

      5
      NumberOfOpenCreditLinesAndLoans
      -0.003472

      3
      DebtRatio
      -0.000017

      0
      RevolvingUtilizationOfUnsecuredLines
      -0.000008

      4
      MonthlyIncome
      -0.000004

      8
      NumberOfTime60-89DaysPastDueNotWorse
      0.000890

      7
      NumberRealEstateLoansOrLines
      0.007461

      6
      NumberOfTimes90DaysLate
      0.014574

      2
      NumberOfTime30-59DaysPastDueNotWorse
      0.023416

      9
      NumberOfDependents
      0.057230

## Random forest

```python
rf_model = rf_pipeline.stages[-1]
```

```python
pd.DataFrame(list(zip(credit_score_df.columns[2:], rf_model.featureImportances.toArray())),
            columns = ['column', 'weight']).sort_values('weight')
```

      column
      weight

      9
      NumberOfDependents
      0.002222

      7
      NumberRealEstateLoansOrLines
      0.006340

      4
      MonthlyIncome
      0.006468

      5
      NumberOfOpenCreditLinesAndLoans
      0.008131

      3
      DebtRatio
      0.014706

      1
      age
      0.014848

      2
      NumberOfTime30-59DaysPastDueNotWorse
      0.150698

      8
      NumberOfTime60-89DaysPastDueNotWorse
      0.207864

      0
      RevolvingUtilizationOfUnsecuredLines
      0.226564

      6
      NumberOfTimes90DaysLate
      0.362159

Let's take a look at one tree

```python
len(rf_model.trees)
```

    20

```python
print(rf_model.trees[0].toDebugString)
```

    DecisionTreeClassificationModel (uid=dtc_0197d19de214) of depth 5 with 63 nodes
      If (feature 8 <= 0.5)
       If (feature 0 <= 0.760372257)
        If (feature 2 <= 1.5)
         If (feature 6 <= 0.5)
          If (feature 2 <= 0.5)
           Predict: 0.0
          Else (feature 2 > 0.5)
           Predict: 0.0
         Else (feature 6 > 0.5)
          If (feature 0 <= 0.3434886975)
           Predict: 0.0
          Else (feature 0 > 0.3434886975)
           Predict: 0.0
        Else (feature 2 > 1.5)
         If (feature 7 <= 6.5)
          If (feature 6 <= 0.5)
           Predict: 0.0
          Else (feature 6 > 0.5)
           Predict: 0.0
         Else (feature 7 > 6.5)
          If (feature 2 <= 2.5)
           Predict: 0.0
          Else (feature 2 > 2.5)
           Predict: 1.0
       Else (feature 0 > 0.760372257)
        If (feature 6 <= 0.5)
         If (feature 2 <= 1.5)
          If (feature 2 <= 0.5)
           Predict: 0.0
          Else (feature 2 > 0.5)
           Predict: 0.0
         Else (feature 2 > 1.5)
          If (feature 3 <= 0.52000446)
           Predict: 0.0
          Else (feature 3 > 0.52000446)
           Predict: 0.0
        Else (feature 6 > 0.5)
         If (feature 6 <= 2.5)
          If (feature 0 <= 0.9999999500000001)
           Predict: 0.0
          Else (feature 0 > 0.9999999500000001)
           Predict: 1.0
         Else (feature 6 > 2.5)
          If (feature 3 <= 0.102934254)
           Predict: 1.0
          Else (feature 3 > 0.102934254)
           Predict: 1.0
      Else (feature 8 > 0.5)
       If (feature 6 <= 0.5)
        If (feature 2 <= 2.5)
         If (feature 3 <= 0.52000446)
          If (feature 9 <= 4.5)
           Predict: 0.0
          Else (feature 9 > 4.5)
           Predict: 1.0
         Else (feature 3 > 0.52000446)
          If (feature 2 <= 0.5)
           Predict: 0.0
          Else (feature 2 > 0.5)
           Predict: 0.0
        Else (feature 2 > 2.5)
         If (feature 3 <= 1.5929396905000002)
          If (feature 8 <= 4.5)
           Predict: 0.0
          Else (feature 8 > 4.5)
           Predict: 1.0
         Else (feature 3 > 1.5929396905000002)
          If (feature 4 <= 887.5)
           Predict: 0.0
          Else (feature 4 > 887.5)
           Predict: 1.0
       Else (feature 6 > 0.5)
        If (feature 2 <= 0.5)
         If (feature 0 <= 0.0029947805)
          If (feature 1 <= 53.5)
           Predict: 1.0
          Else (feature 1 > 53.5)
           Predict: 0.0
         Else (feature 0 > 0.0029947805)
          If (feature 3 <= 0.043076442)
           Predict: 0.0
          Else (feature 3 > 0.043076442)
           Predict: 0.0
        Else (feature 2 > 0.5)
         If (feature 0 <= 0.6742148045)
          If (feature 1 <= 31.5)
           Predict: 1.0
          Else (feature 1 > 31.5)
           Predict: 0.0
         Else (feature 0 > 0.6742148045)
          If (feature 5 <= 13.5)
           Predict: 1.0
          Else (feature 5 > 13.5)
           Predict: 1.0


## 01강. 선형성 정의 및 1차 연립방정식 의미

### 00. Linearity 선형성

- - -

- Linearity 행렬로 표현되는 것이 기본적으로 선형성을 만족해야 함
    - $f(x)$, x라는 변수에 대해서 function/operation 을 가지느냐

#### 조건 2가지를 만족할 때 Linearity 성립
- Superposition(중첩의 원리) : $f(x_{1}+x_{2}) = f(x_{1}) + f(x_{2})$
- Homogeniety $f(ax) = af(x)$ , a는 constant
    - $f(a_{1}x_{1}+a_{2}x_{2})=a_{1}f(x_{1})+a_{2}f(x_{2})$ : 이 조건을 만족할 때 선형성이 있다고 함, 이런 것들 선형대수에서 다룬다.

- $ y = mx = f(x) $ : 위 두 조건을 만족하며 linear 하다
- $ y = mx + n $ : 원점을 지나지 않는 직선은 x와 y사이에 linearity 가 존재하지 않음, linear equation이라고 다 linearity를 가지는 것은 아니다. 하지만 변화량에는 선형성이 존재

#### operation의 linearity : differentiation, integration
 - $x_{1}(t), x_{2}(t) $ 에 대한 diff, inte의 선형성 알아보자 $\frac{d}{dt}(a_{1}x_{1}(t)+a_{2}x_{2}(t)) = a_{1}\frac{d}{dt}x_{1}(t)+a_{2}\frac{d}{dt}x_{2}(t)$ : linear 하다
    - integration 해도 마찬가지로 만족된다.
    - 미분 적분 linearity 가지기 때문에 매트릭스 폼으로 바꿔서 연산 수행 가능

#### matrix의 linearity : 행렬 A에 벡터 $x_{1}, x_{2}$을 곱할 때 새로운 형태의 벡터 y가 나오는데
- $A(a_{1}x_{1}+a_{2}x_{2}) = a_{1}Ax_{1} + a_{2}Ax_{2}$

### Before getting started : Basic Notations of Matrix
- - -

Vector v = (a, b, c) row vec 아니라 col vec 으로 쓰자

\begin{align}
    v = \begin{bmatrix}
           a \\
           b \\
           c
         \end{bmatrix}
  \end{align}

- transpose(전치) : row와 column의 값을 interchange하는 것

### 01. Linear Combination(선형결합)
- - -

\begin{align}
    v = \begin{bmatrix}
           a_{1} \\
           b_{1} \\
           c_{1}
         \end{bmatrix}
  \end{align}

\begin{align}
    w = \begin{bmatrix}
           a_{2} \\
           b_{2} \\
           c_{2}
         \end{bmatrix}
  \end{align}

$\alpha v + \beta w$를 계산한다

possible to express linear combination with multiplication of matrix and vector

\begin{align}
    \begin{bmatrix}
           v \ w
         \end{bmatrix} =
    \begin{bmatrix}
           a_{1} \ a_{2} \\
            b_{1} \ b_{2} \\
            c_{1} \ c_{2} \\
         \end{bmatrix}
  \end{align}

\begin{align}
  \begin{bmatrix}
           a_{1} \ a_{2} \\
            b_{1} \ b_{2} \\
            c_{1} \ c_{2} \\
    \end{bmatrix}
    \begin{bmatrix}
            \alpha \\
            \beta \\
    \end{bmatrix}
\end{align}

### 02. What is Matrix?
- - -

- matrix is a set of number arranged in m x n (rows x columns)

- matrix calculation : + - * (by using inverse matrix)

- identitiy matrix : I

- inverse matrix : right and left inverse
    - 2x2 matrix can be defined by using determinant! $\frac{1}{ad-bc}$

### 03. What is Vector?
- - -

\begin{align}
    v = \begin{bmatrix}
           a \\
           b \\
           c
         \end{bmatrix}
\end{align}

geometrically addition of vector is drawn like parallelogram

subtraction of vector $v_{1}-v_{2}$ is defined as a arrow from $v_{2}$ to $v_{1}$

- inner product(내적)

$|v_{1}||v_{2}| cos\theta$

### Chap1. Gauss Elimination

- - -

how to solve linear system sgn(부호함수) : 가우스 소거법으로 풀자

\begin{align}
  \begin{bmatrix}
           1 \ \ 2 \\
            4 \ \ 5 \\
    \end{bmatrix}
    \begin{bmatrix}
            x \\
            y \\
    \end{bmatrix} =
    \begin{bmatrix}
            3 \\
            6 \\
    \end{bmatrix}
\end{align}

geometically meaning of solution is intersection of two lines

however, question could be solved with linear combination of two column vectors as well.

#### For 3D  4D vector

row form - intersection

column form - linear combination of column vectors


# **JIVE: Joint and Individual Variation Explained**

JIVE (Joint and Individual Variation Explained) is a dimensional reduction algorithm that can be used when there are multiple data matrices (data blocks). The multiple data block setting means there are $K$ different data matrices, with the same number of observations $n$ and (possibly) different numbers of variables ($d_1, \dots, d_k$). JIVE finds modes of variation which are common (joint) to all $K$ data blocks and modes of individual variation which are specific to each block. For a detailed discussion of JIVE see [Angle-Based Joint and Individual Variation Explained](https://arxiv.org/pdf/1704.02060.pdf).[^1]

For a concrete example, consider a two block example from a medical study. Suppose there are $n=500$ patients (observations). For each patient we have $d_1 = 100$ bio-medical variables (e.g. hit, weight, etc). Additionally we have $d_2 = 10,000$ gene expression measurements for each patient.

## **The JIVE decomposition**

Suppose we have $K$ data data matrices (blocks) with the same number of observations, but possibly different numbers of variables; in particular let $X^{(1)}, \dots, X^{(K)}$ where $X^{(k)} \in \mathbb{R}^{n \times d_k}$. JIVE will then decompose each matrix into three components: joint signal, individual signal and noise

\begin{equation}
X^{(k)} = J^{(k)} + I^{(k)} + E^{(k)}
\end{equation}

where $J^{(k)}$ is the joint signal estimate, $I^{(k)}$ is the individual signal estimate and $E^{(k)}$ is the noise estimate (each of these matrices must the same shape as the original data block: $\mathbb{R}^{n \times d_k}$). Note: **we assume each data matrix** $X^{(k)}$ **has been column mean centered**.

The matrices satisfy the following constraints:

1. The joint matrices have a common rank: $rk(J^{(k)}) = r_{joint}$ for $k=1, \dots, K$.
2. The individual matrices have block specific ranks $rk(I^{(k)}) = r_{individual}^{(k)}$.
3. The columns of the joint matrices share a common space called the joint score space (a subspace of $\mathbb{R}^n$); in particular the $\text{col-span}(J^{(1)}) = \dots = \text{col-span}(J^{(K)})$ (hence the name joint).
4. Each individual spaces score subspace (of $\mathbb{R}^n$) is orthogonal to the the joint space; in particular $\text{col-span}(J^{(k)}) \perp \text{col-span}(I^{(k)})$ for $k=1, \dots, K$.

Note that JIVE may be more natural if we think about data matrices subspaces of $\mathbb{R}^n$ (the score space perspective). Typically we think of a data matrix as $n$ points in $\mathbb{R}^d$. The score space perspective views a data matrix as $d$ vectors in $\mathbb{R}^n$ (or rather the span of these vectors). One important consequence of this perspective is that it makes sense to related the data blocks in score space (e.g. as subspaces of $\mathbb{R}^n$) since they share observtions.

## Quantities of interest

There are a number of potential quantities of interest depending on the application. For example the user may be interested in the full matrices $J^{(k)}$ and/or $I^{(k)}$. By construction these matrices are not full rank and we may also be interested in their singular value decomposition which we define as

\begin{align}
& U^{(k)}_{joint}, D^{(k)}_{joint}, V^{(k)}_{joint} = \text{rank } r_{joint} \text{ SVD of } J^{(k)} \\
& U^{(k)}_{individual}, D^{(k)}_{individual}, V^{(k)}_{individual} = \text{rank } r_{individual}^{{k}} \text{ SVD of } I^{(k)}
\end{align}

One additional quantity of interest is $U_{joint} \in \mathbb{R}^{n \times r_{joint}}$ which is an orthogonal basis of $\text{col-span}(J^{(k)})$. This matrix is produced from an intermediate JIVE computation.

## **PCA analogy**
We give a brief discussion of the PCA/SVD decomposition (assuming the reading is already familiar).

#### Basic decomposition
Suppose we have a data matrix $X \in \mathbb{n \times d}$. Assume that $X$ has been column mean centered and consider the SVD decomposition (this is PCA since we have mean centered the data):

\begin{equation}
X = U D V^T.
\end{equation}
where $U \in \mathbb{R}^{n \times m}$, $D \in \mathbb{R}^{m \times m}$ is diagonal, and $V \in \mathbb{R}^{d \times m}$ with $m = min(n, d)$. Note $U^TU = V^TV = I_{m \times m}$.

Suppose we have decided to use a rank $r$ approximation. We can then decompose $X$ into a signal matrix ($A$) and an noise matrix ($E$)

\begin{equation}
X = A + E,
\end{equation}
where $A$ is the rank $r$ SVD approximation of $X$ i.e.
\begin{align}
A := & U_{:, 1:r} D_{1:r, 1:r} V_{:, 1:r}^T \\
 = & \widetilde{U}, \widetilde{D} \widetilde{V}^T
\end{align}
The notation $U_{:, 1:r} \in \mathbb{R}^{n \times r}$ means the first $r$ columns of $U$. Similarly we can see the error matrix is $E :=U_{:, r+1:n} D_{r+1:m, r_1:m} V_{:, r+1:d}^T$.

#### Quantities of interest

There are many ways to use a PCA/SVD decomposition. Some common quantities of interest include

- The normalized scores: $\widetilde{U} \in \mathbb{R}^{n \times r}$
- The unnormalized scores: $\widetilde{U}\widetilde{D} \in \mathbb{R}^{n \times r}$
- The loadings: $\widetilde{V} \in \mathbb{R}^{d \times r}$
- The full signal approximation: $A \in \mathbb{R}^{n \times d}$

#### Scores and loadings

For both PCA and JIVE we use the notation $U$ (scores) and $V$ (loadings). These show up in several places.

We refer to all $U \in \mathbb{R}^{n \times r}$ matrices as scores. We can view the $n$ rows of $U$ as representing the $n$ data points with $r$ derived variables (put differently, columns of $U$ are $r$ derived variables). The columns of $U$ are orthonormal: $U^TU = I_{r \times r}$.

Sometimes we may want $UD$ i.e scale the columns of $U$ by $D$ (the columns are still orthogonal). The can be useful when we want to represent the original data by $r$ variables. We refer to $UD$ as unnormalized scores.

We refer to all $V\in \mathbb{R}^{d \times r}$ matrices as loadings[^2]. The j$th$ column of $V$ gives the linear combination of the original $d$ variables which is equal to the j$th$ unnormalized scores (j$th$ column of $UD$). Equivalently, if we project the $n$ data points (rows of $X$) onto the j$th$ column of $V$ we get the j$th$ unnormalized scores.

The typical geometric perspective of PCA is that the scores represent $r$ new derived variables. For example, if $r = 2$ we can look at a scatter plot that gives a two dimensional approximation of the data. In other words, the rows of the scores matrix are $n$ data points living in $\mathbb{R}^r$.

An alternative geometric perspective is the $r$ columns of the scores matrix are vectors living in $\mathbb{R}^n$. The original $d$ variables span a subspace of $\mathbb{R}^n$ given by $\text{col-span}(X)$. The scores then span a lower dimensional subspace of $\mathbb{R}^n$ that approximates $\text{col-span}(X)$.

The first perspective says PCA finds a lower dimensional approximation to a subspace in $\mathbb{R}^d$ (spanned by the $n$ data points). The second perspective says PCA finds a lower dimensional approximation to a subspace in $\mathbb{R}^n$ (spanned by the $d$ data points).

## **JIVE operating in score space**

For a data matrix $X$ let's call the span of the variables (columns) the *score subpace*, $\text{col-span}(X) \subset \mathbb{R}^n$. Typically we think of a data matrix as $n$ points in $\mathbb{R}^d$. The score space perspective reverses this and says a data matrix is $d$ points in $\mathbb{R}^n$. When thinking in the score space it's common to consider about subspaces i.e. the span of the $d$ variables in $\mathbb{R}^n$. In other words, if two data matrices have the same column span then their score subspaces are the same[^3].

JIVE partitions the score space of each data matrix into three subspaces: joint, individual and noise. The joint score subspace for each data block is the same. The individual score subspace, however, is (possibly) different for each of the $K$ blocks. The k$th$ block's individual score subspace is orthogonal to the joint score subspace. Recall that the $K$ data matrices have the same number of observations ($n$) so it makes sense to think about how the data matrices relate to each other in score space.

PCA partitions the score space into two subspaces: signal and noise (see above). For JIVE we might combine the joint and individual score subspaces and call this the signal score subspace.

# Footnotes
[^1]: Note this paper calls the algorithm AJIVE (angle based JIVE) however, we simply use JIVE. Additionally, the paper uses columns as observations in data matrices where as we use rows as observations.

[^2]: For PCA we used tildes (e.g. $\widetilde{U}$) to denote the "partial" SVD approximation however for the final JIVE decomposition we do not use tildes. This is intentional since for JIVE the SVD comes from the $I$ and $J$ matrices which are exactly rank $r$. Therefore we view this SVD as the "full" SVD.

[^3]: This might remind the reader of TODO