# Properties of Pure Fluids

```python
import numpy as np
import cantera as ct
from pint import UnitRegistry
ureg = UnitRegistry()
Q_ = ureg.Quantity
```

We can obtain critical properties of fluids using Cantera:

```python
f = ct.Nitrogen()
print(f'Critical temperature: {f.critical_temperature: .2f} K')
print(f'Critical pressure: {f.critical_pressure: .2e} Pa')

density_crit = f.critical_density
mol_weight = f.mean_molecular_weight
z_crit = (
    f.critical_pressure * mol_weight /
    (density_crit * ct.gas_constant * f.critical_temperature)
    )
print(f'Critical compressibility factor: {z_crit: .2f}')
```

    Critical temperature:  126.20 K
    Critical pressure:  3.40e+06 Pa
    Critical compressibility factor:  0.29

## van der Waals equation of state

The van der Waals equation of state is

$$
P = \frac{RT}{v - b} - \frac{a}{v^2} \;,
$$

where $a$ and $b$ are substance-specific constants.

We can find these constants by applying the contraints on equations of state along the critical isotherms:

$$
\left( \frac{\partial P}{\partial v} \right)_T = 0 \\
\left( \frac{\partial^2 P}{\partial v^2} \right)_T = 0 \;,
$$

which both apply at the critical point. These will give us closed-form expressions for $a$ and $b$ based on the critical properties.

Let's use SymPy to help us find these expressions, by taking these derivatives and applying the constraints at the critical point:

```python
import sympy
sympy.init_printing(use_latex='mathjax')

R, Tc, vc, Pc, a, b = sympy.symbols('R, T_c, v_c, P_c, a, b', real=True)

equation_state = (R*Tc)/(vc - b) - (a/vc**2)

# first derivative with respect to v
first_deriv = sympy.diff(equation_state, vc)
display(sympy.Eq(first_deriv, 0))

second_deriv = sympy.diff(first_deriv, vc)
display(sympy.Eq(second_deriv, 0))
```

$\displaystyle - \frac{R T_{c}}{\left(- b + v_{c}\right)^{2}} + \frac{2 a}{v_{c}^{3}} = 0$

$\displaystyle \frac{2 R T_{c}}{\left(- b + v_{c}\right)^{3}} - \frac{6 a}{v_{c}^{4}} = 0$

Then, we can solve this system of two equations for our two unknowns:

```python
sol = sympy.solve((
    first_deriv,
    second_deriv,
    ), (a, b), dict=True)

display(sol[0])

a_expr = sol[0][a]
b_expr = sol[0][b]
```

$\displaystyle \left\{ a : \frac{9 R T_{c} v_{c}}{8}, \  b : \frac{v_{c}}{3}\right\}$

This gives us the coefficients $a$ and $b$ as a function of $T_{\text{crit}}$, $P_{\text{crit}}$, and $R$. However, while the critical temperature can be measured experimentally *relatively* easily, the critical specific volume is harder to measure accurately.

So, we can introduce the equation of state applied at the critical point,

$$
P_{\text{crit}} = \frac{R T_{\text{crit}}}{v_{\text{crit}} - b} - \frac{a}{v_{\text{crit}}^2} \;,
$$

and then solve the system of three equations for $a$, $b$, and $v_{\text{crit}}$ to eliminate the critical specific volume:

```python
sol = sympy.solve((
    a - a_expr, b - b_expr, Pc - ((R*Tc / (vc-b)) - (a / vc**2))
    ), (a, b, vc), dict=True)
display(sol[0])
```

$\displaystyle \left\{ a : \frac{27 R^{2} T_{c}^{2}}{64 P_{c}}, \  b : \frac{R T_{c}}{8 P_{c}}, \  v_{c} : \frac{3 R T_{c}}{8 P_{c}}\right\}$

Thus, we can now determine the coefficients $a$ and $b$ based on measured values of the critical temperature and pressure.

We can now use the equation of state to get $P = f(T,v)$, but what about determining $v$ based on $T$ and $P$? It turns out that the van der Waals equation of state gives a cubic function of $v$, which when expressed in terms of reduced properties is
\begin{equation}
v_r^3 - \left( \frac{8}{3} \frac{T_r}{P_r} + \frac{1}{3} \right) + \frac{3}{P_r} v_r - \frac{1}{P_r} = 0
\end{equation}
This means that we need to find the roots of this cubic polynomial to get specific volume. As a consequence, there will be three roots and potential solutions for $v_r$.
At minimum, we should only consider real roots; imaginary roots are not physically meaningful.

First, consider when $T_r > 1.0$, such as for $T_r = 2.5$ and $P_r = 2.0$:

```python
temp_r = 2.5
pres_r = 2.0
coeffs = [
    1.0,
    -((8.0*temp_r / (3.0*pres_r)) + (1.0 / 3.0)),
    3.0 / pres_r,
    -1.0 / pres_r
    ]
roots = np.roots(coeffs)
print(f'Roots: {roots}')
# get only real roots
roots = roots.real[np.abs(roots.imag)<1e-5]

print(f'Number of real roots: {len(roots)}')

print(f'T_r = {temp_r: .2f}')
print(f'P_r = {pres_r: .2f}')

reduced_volumes = ','.join([f'{vol_r: .3f}' for vol_r in roots])
print(f'v_r = {reduced_volumes}')
```

    Roots: [3.25277894+0.j         0.20694386+0.33299993j 0.20694386-0.33299993j]
    Number of real roots: 1
    T_r =  2.50
    P_r =  2.00
    v_r =  3.253

So, for temperatures above the critical temperature, there will be only one real value for reduced specific volume.

But, if $T_r < 1$, we can get three real roots:

```python
temp_r = 0.90
pres_r = 0.61
coeffs = [
    1.0,
    -((8.0*temp_r / (3.0*pres_r)) + (1.0 / 3.0)),
    3.0 / pres_r,
    -1.0 / pres_r
    ]
roots = np.roots(coeffs)
roots = roots.real[np.abs(roots.imag)<1e-5]

print(f'Number of real roots: {len(roots)}')

print(f'T_r = {temp_r: .2f}')
print(f'P_r = {pres_r: .2f}')

reduced_volumes = ','.join([f'{vol_r: .3f}' for vol_r in roots])
print(f'v_r = {reduced_volumes}')
```

    Number of real roots: 3
    T_r =  0.90
    P_r =  0.61
    v_r =  2.640, 1.017, 0.610

In this case, we need to apply some additional knowledge to understand these roots and identify any non-physical values, such as considering the sign of $\frac{\partial P_r}{\partial v_r}$ for these points.


# Ecuación de difusión en 1-D

La ecuación de difusión unidimensional está dada por:

$$\frac{\partial u}{\partial t}= \nu \frac{\partial^2 u}{\partial x^2}$$

La derivada de segundo orden se puede representar geométricamente como la línea tangente a la curva dada por la primera derivada. Discretizaremos la derivada de segundo orden con un esquema de Diferencia Central: una combinación de Diferencia hacia adelante y Diferencia hacia atrás de la primera derivada. Considere la expansión de Taylor de $ u_{i + 1} $ y $ u_{i-1} $ alrededor de $ u_i $:

$u_{i+1} = u_i + \Delta x \frac{\partial u}{\partial x}\bigg|_i + \frac{\Delta x^2}{2} \frac{\partial ^2 u}{\partial x^2}\bigg|_i + \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\bigg|_i + O(\Delta x^4)$

$u_{i-1} = u_i - \Delta x \frac{\partial u}{\partial x}\bigg|_i + \frac{\Delta x^2}{2} \frac{\partial ^2 u}{\partial x^2}\bigg|_i - \frac{\Delta x^3}{3!} \frac{\partial ^3 u}{\partial x^3}\bigg|_i + O(\Delta x^4)$

Si sumamos estas dos expansiones, puede ver que los términos de las derivadas impares se cancelarán entre sí. Si descuidamos cualquier término de $ O (\Delta x ^ 4) $ o superior (y en realidad, esos son muy pequeños), entonces podemos reorganizar la suma de estas dos expansiones para resolver nuestra segunda derivada.

$$u_{i+1} + u_{i-1} = 2u_i+\Delta x^2 \frac{\partial ^2 u}{\partial x^2}\bigg|_i + O(\Delta x^4)$$
Así,
$$\frac{\partial ^2 u}{\partial x^2}=\frac{u_{i+1}-2u_{i}+u_{i-1}}{\Delta x^2} + O(\Delta x^4)$$

Así, reemplazando en la ecuación de Difusión

$$\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=\nu\frac{u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n}}{\Delta x^2}$$

Como antes, notamos que una vez que tenemos una condición inicial, la única incógnita es $ u_ {i} ^ {n + 1} $, por lo que reorganizamos la ecuación resolviendo nuestra incógnita:

$$u_{i}^{n+1}=u_{i}^{n}+\frac{\nu\Delta t}{\Delta x^2}(u_{i+1}^{n}-2u_{i}^{n}+u_{i-1}^{n})$$

```python
import numpy as np
import matplotlib.pyplot as plt
import time, sys
%matplotlib inline
```

# Retomemos las funciones para el dato inicial.

```python
nx = 41
dx = 2 / (nx - 1)
nt = 20    #el número de pasos de tiempo que queremos calcular
nu = 0.3   #el valor de viscocidad
sigma = .2
dt = sigma * dx**2 / nu #note la diferencia...
x=np.arange(0.0,2+dx,dx)
```

```python
dt*nt #es el tiempo final, al terminar la iteración
```

    0.03333333333333335

```python
def f1(nx):
    #dx = 2 / (nx-1)
    u = np.ones(nx)      #numpy función ones()
    u[int(.5 / dx):int(1 / dx + 1)] = 2  #configurar u = 2 entre 0.5 y 1 y u = 1 para todo lo demás según nuestra CI
    #print(u)
    return u

def f1_var(x):
    u=np.piecewise(x,[x<0.5,np.abs(x-0.75)<=0.25,x>1],[lambda x:1,lambda x: 2,lambda x:1])
    return u

def f2(nx):
    #dx = 2 / (nx-1)
    L=2
    n=4
    Fi=0.2#ángulo de fase temporal
    c=10 #velocidad de la onda
    A=0.5#amplitud máxima, relacionada con la intensidad del sonido
    t=0.18 #instantánea en el tiempo t segundos
    x=np.arange(0.0,L+dx,dx)
    u=A*np.sin(n*np.pi*c*0.005*t/L+Fi)*np.sin(n*np.pi*x/L)#aquí definimos la función
    return u

def mostrar_imagen(u):
    plt.plot(np.linspace(0, 2, nx), u);
```

# Discretizamos la ecuación de Difusión ED

```python
def ED(u):
    for n in range(nt):  #iterando en el tiempo
        un = u.copy()
        for i in range(1, nx - 1): #iterando en el espacio
            u[i] = un[i] + nu * dt / dx**2 * (un[i+1] - 2 * un[i] + un[i-1])
    return u
```

```python
u=ED(f1_var(x))
mostrar_imagen(u)
```

```python
u=ED(f2(nx))
mostrar_imagen(u)
```

```python
nx = 41
dx = 2 / (nx - 1)
nt = 20    #el número de pasos de tiempo que queremos calcular
nu = 0.3   #el valor de viscocidad
sigma = .2
dt = sigma * dx**2 / nu #note la diferencia...
x=np.arange(0.0,2+dx,dx)

plt.figure(figsize=(18,12))
#como colocar un título general a un paquete

plt.subplot(2, 2, 1)
plt.plot(x,f1(nx),label=f' Velocidad inicial para f1')
plt.legend(frameon=False)
plt.subplot(2, 2, 2)
plt.plot(x,ED(f1(nx)),label=f'Vel en t=0.5 para f1')
plt.legend(frameon=False)
plt.subplot(2, 2, 3)
plt.plot(x,f2(nx),label=f' Velocidad inicial para f2')
plt.legend(frameon=False)
plt.subplot(2, 2, 4)
plt.plot(x,ED(f2(nx)),label=f'Vel en t=0.5 para f2')
plt.legend(frameon=False)
plt.show()
```

# Ecuación de Burgers

La ecuación de Burgues (EB) en una dimensión espacial, es una combinación de la ecuación de convección no lineal y la ecuación de difusión. Vamos a detenernos y estudiar un poco esta ecuación por lo sorprendente que es y lo mucho que podemos aprender de esta pequeña ecaución.

$$\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} = \nu \frac{\partial ^2u}{\partial x^2}$$

 Usando los pasos realizados en la discretización de la ECnL y la ED obtenemos:

 $$\frac{u_i^{n+1}-u_i^n}{\Delta t} + u_i^n \frac{u_i^n - u_{i-1}^n}{\Delta x} = \nu \frac{u_{i+1}^n - 2u_i^n + u_{i-1}^n}{\Delta x^2}$$

De donde,

$$u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)$$

Para examinar algunas propiedades interesantes de la ecuación de Burgers, es útil usar diferentes condiciones iniciales y de contorno de las que hemos estado usando para los pasos anteriores.

Nuestra condición inicial para este problema será:

\begin{eqnarray}
u &=& -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \\\
\phi &=& \exp \bigg(\frac{-x^2}{4 \nu} \bigg) + \exp \bigg(\frac{-(x-2 \pi)^2}{4 \nu} \bigg)
\end{eqnarray}

Cuya solución analítica es dada por( Usando el método de separación de variables)
\begin{eqnarray}
u &=& -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \\\
\phi &=& \exp \bigg(\frac{-(x-4t)^2}{4 \nu (t+1)} \bigg) + \exp \bigg(\frac{-(x-4t -2 \pi)^2}{4 \nu(t+1)} \bigg)
\end{eqnarray}

Además, usaremos una condición de frontera (problema tipo Neumann)

$$u(0) = u(2\pi).$$

La condición inicial que estamos usando para la ecuación de Burgers puede ser un poco difícil de evaluar a mano. La derivada $\frac{\partial \phi}{\partial x}$ no es demasiado difícil, pero sería fácil dejar caer un signo u olvidar un factor de $ x $ en alguna parte, así que vamos a usar SymPy para ayudarnos.

# [SymPy](http://sympy.org/en/)

SymPy es la biblioteca matemática simbólica de Python. Tiene muchas de las mismas funciones matemáticas simbólicas que Mathematica con el beneficio adicional de que podemos traducir fácilmente sus resultados a nuestros cálculos de Python (también es gratuito y de código abierto).

```python
import sympy
from sympy import init_printing
init_printing(use_latex=True) #salidas renderizadas usando Latex
```

Comience configurando variables simbólicas para las tres variables en nuestra condición inicial y luego escriba la ecuación completa para $ \phi $. Deberíamos obtener una versión bien renderizada de nuestra ecuación $ \phi $.

```python
x, nu, t = sympy.symbols('x nu t')
phi = (sympy.exp(-(x - 4 * t)**2 / (4 * nu * (t + 1))) +
       sympy.exp(-(x - 4 * t - 2 * sympy.pi)**2 / (4 * nu * (t + 1))))
phi
```

```python
phiprimex = phi.diff(x)
phiprimex
```

```python
phiprimexx = phiprimex.diff(x)
phiprimexx
```

```python
phiprimet = phi.diff(t)
phiprimet
```

Ahora que tenemos la versión Pythonic de nuestra derivada, podemos terminar de escribir la ecuación de la condición inicial completa y luego traducirla a una expresión Python utilizable. Para esto, usaremos la función *lambdify*, que toma una ecuación simbólica SymPy y la convierte en una función invocable.

```python
from sympy.utilities.lambdify import lambdify

u = -2 * nu * (phiprimex / phi) + 4
print(u)
```

    -2*nu*(-(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 4*pi)*exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)))/(exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1))) + exp(-(-4*t + x)**2/(4*nu*(t + 1)))) + 4

```python
v = u.diff(t) + u * u.diff(x) - nu * u.diff(x,x)
```

```python
vfunc = lambdify((t, x, nu), v)
print(vfunc(1, 4, 3))
```

    0.0

## Lambdify
Para lambdify esta expresión en una función utilizable, le decimos a lambdify qué variables solicitar y la función a la que queremos conectarlas.

```python
ufunc = lambdify((t, x, nu), u)
print(ufunc(1, 4, 3))
```

    3.49170664206445

Ahora que tenemos configuradas las condiciones iniciales, podemos continuar y terminar de configurar el problema. Podemos generar la gráfica de la condición inicial usando nuestra función lambdify

```python
nx = 101
nt = 100
dx = 2 * np.pi / (nx - 1)
nu = .07
dt = dx * nu
x = np.linspace(0, 2 * np.pi, nx)
def f3(t):
    x = np.linspace(0, 2 * np.pi, nx)
    un = np.empty(nx)
    u = np.asarray([ufunc(t, x0, nu) for x0 in x])
    return u
np.shape(f3(0))
```

```python
np.shape(x)
```

```python
u = f3(0)
plt.figure(figsize=(11, 7), dpi=100)
plt.plot(x, u, marker='o', lw=2)
plt.xlim([0, 2 * np.pi])
plt.ylim([0, 10]);
```

Definitivamente, esta no es la función de sombrero con la que hemos estado tratando hasta ahora. Lo llamamos "función de diente de sierra". Sigamos adelante y veamos qué sucede.

## Condiciones de frontera periódicas

Una de las grandes diferencias entre lo que habíamos hecho y este problema  es el uso de condiciones de contorno *periódicas*. Si experimenta un poco con los lo hecho antes y hace que la simulación se ejecute más tiempo (aumentando `nt`), notará que la onda seguirá moviéndose hacia la derecha hasta que ya ni siquiera aparezca en la gráfica.

Con condiciones de contorno periódicas, cuando un punto llega al lado derecho del marco, **se envuelve** de nuevo al frente del marco.

Recuerde la discretización que resolvimos
$$u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)$$

¿Qué *significa* $ u_{i + 1} ^ n $  cuando $ i $ ya está al final del marco?

Piense en esto por un minuto antes de continuar.

```python
u = f3(0) #Volvemos a crear la condición inicial
def EB(u):
    for n in range(nt):
        un = u.copy()
        for i in range(nx-1):
            u[i] = un[i] - un[i] * dt / dx *(un[i] - un[i-1]) + nu * dt / dx**2 *\
                (un[i+1] - 2 * un[i] + un[i-1])
        u[-1] = u[0]
    return u

u_analitica = np.asarray([ufunc(nt * dt, xi, nu) for xi in x])
```

```python
u = f3(0) #Volvemos a crear la condición inicial
def EB_profe(u):
    for n in range(nt):
        # el equivalente de np.roll es pandas es pd.DataFrame.shift (me parece)
        # En ese caso no asume periodicidad !
        u_der = np.roll(u,-1)
        u_izq = np.roll(u,1)
        u = u - u *dt / dx * (u -u_izq) + nu * dt / dx**2 * (u_izq - 2*u +u_der)
    return u

```

```python
plt.figure(figsize=(11, 7), dpi=100)
plt.plot(x,u, marker='o', lw=2, label='Inicial')
plt.plot(x,EB(u), marker='o', lw=2, label='Computacional')
plt.plot(x, u_analitica, label='Analítica')
plt.xlim([0, 2 * np.pi])
plt.ylim([0, 10])
plt.legend(frameon = False);
```

# Jugando un poco con los parametros

```python
nx = 301 # tome diferentes valores y si vale 501?
nt = 1000 # tome diferentes valores y si vale 500?
dx = 2 * np.pi / (nx - 1)
nu = .07
dt = dx * nu
x = np.linspace(0, 2 * np.pi, nx)
u = f3(0) #Volvemos a crear la condición inicial
u_analytical = np.asarray([ufunc(nt * dt, xi, nu) for xi in x])
plt.figure(figsize=(11, 7), dpi=100)
plt.plot(x,u, marker='o', lw=2, label='Inicial')
plt.plot(x,EB(u), marker='o', lw=2, label='Computacional')
plt.plot(x, u_analytical, label='Analítica')
plt.xlim([0, 2 * np.pi])
plt.ylim([0, 10])
plt.legend(frameon = False);
```

```python

```


# Interact Exercise 6

## Imports

Put the standard imports for Matplotlib, Numpy and the IPython widgets in the following cell.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

```python
from IPython.display import Image
from IPython.html.widgets import interact, interactive, fixed
```

## Exploring the Fermi distribution

In quantum statistics, the [Fermi-Dirac](http://en.wikipedia.org/wiki/Fermi%E2%80%93Dirac_statistics) distribution is related to the probability that a particle will be in a quantum state with energy $\epsilon$. The equation for the distribution $F(\epsilon)$ is:

```python
Image('fermidist.png')
```

In this equation:

* $\epsilon$ is the single particle energy.
* $\mu$ is the chemical potential, which is related to the total number of particles.
* $k$ is the Boltzmann constant.
* $T$ is the temperature in Kelvin.

In the cell below, typeset this equation using LaTeX:

\begin{equation}
F(\epsilon) = {\displaystyle \frac{1}{e^{(\epsilon - \mu)/kT}+1}}
\end{equation}

Define a function `fermidist(energy, mu, kT)` that computes the distribution function for a given value of `energy`, chemical potential `mu` and temperature `kT`. Note here, `kT` is a single variable with units of energy. Make sure your function works with an array and don't use any `for` or `while` loops in your code.

```python
def fermidist(energy, mu, kT):
    """Compute the Fermi distribution at energy, mu and kT."""
    return (np.exp((energy-mu)/kT)+1)**-1
```

```python
assert np.allclose(fermidist(0.5, 1.0, 10.0), 0.51249739648421033)
assert np.allclose(fermidist(np.linspace(0.0,1.0,10), 1.0, 10.0),
    np.array([ 0.52497919,  0.5222076 ,  0.51943465,  0.5166605 ,  0.51388532,
               0.51110928,  0.50833256,  0.50555533,  0.50277775,  0.5       ]))
```

Write a function `plot_fermidist(mu, kT)` that plots the Fermi distribution $F(\epsilon)$ as a function of $\epsilon$ as a line plot for the parameters `mu` and `kT`.

* Use enegies over the range $[0,10.0]$ and a suitable number of points.
* Choose an appropriate x and y limit for your visualization.
* Label your x and y axis and the overall visualization.
* Customize your plot in 3 other ways to make it effective and beautiful.

```python
np.arange(0,10.01,0.01)
```

    array([  0.  ,   0.01,   0.02, ...,   9.98,   9.99,  10.  ])

```python
def plot_fermidist(mu, kT):
    energy=np.arange(0,10.01,0.01)
    plt.figure(figsize=(10,6))
    plt.plot(energy,fermidist(energy,mu,kT))
    plt.tick_params(axis='x', top='off')
    plt.tick_params(axis='y', right='off')
    plt.xlabel('Energy')
    plt.xlim(left=0, right=10)
    plt.ylim(bottom=0.0,top=1.0)
    plt.ylabel('Fermi Distribution')
```

```python
plot_fermidist(4.0, 1.0)
```

```python
assert True # leave this for grading the plot_fermidist function
```

Use `interact` with `plot_fermidist` to explore the distribution:

* For `mu` use a floating point slider over the range $[0.0,5.0]$.
* for `kT` use a floating point slider over the range $[0.1,10.0]$.

```python
interact(plot_fermidist, mu=(0.0,5.0,0.1), kT=(0.1,10.0,0.1));
```

Provide complete sentence answers to the following questions in the cell below:

* What happens when the temperature $kT$ is low?
* What happens when the temperature $kT$ is high?
* What is the effect of changing the chemical potential $\mu$?
* The number of particles in the system are related to the area under this curve. How does the chemical potential affect the number of particles.

Use LaTeX to typeset any mathematical symbols in your answer.

$\bullet$ When $kT$ is low, the Fermi distribution starts off very high, around $1.0$, but then sharply drops off to zero, or near zero, as energy increases.

$\bullet$ When $kT$ is high, the Fermi distribution becomes more linear in its shape.

$\bullet$ Changing the chemical potential of the equation "shifts" the graph left or right as $\mu$ decreases or increases, respectively.

$\bullet$ As the chemical potential increases from $0$ to $5$, the area under the curve inceases as well. This means the particles in the system are being distributed more throughout the energies of the system. However, that is if we hold $kT$ at a constant value. If we increase both $kT$ and $\mu$ together, the area under the curve will change constantly, thus the particles are even more distributed throughout the energies.


# Taylor problem 1.51

This problem attacks the "oscillating skateboard" problem described in Example 1.2 of Taylor.  A Newton's 2nd law analysis leads to the differential equation for the angle $\phi$ in radians:

$
\begin{align}
  \ddot\phi = -\frac{g}{R}\sin\phi
  \;.
\end{align}
$

This is a 2nd order, *nonlinear* differential equation.  We note it is the same equation describing the motion of a simple (undamped, not driven) pendulum.

Problem 1.50 has us solving this equation numerically for particular initial conditions and comparing the plots to the approximate solution based on the small angle approximation for $\sin\phi$.  We'll build up code to find this solution and plot it in steps to illustrate how a notebook evolves.  We don't create the polished version at once!

**Your goal for problem 1.51: Modify the relevant part of this notebook to produce the required figure, print it out, and turn it in with your homework.**

```python
%matplotlib inline
```

```python
import numpy as np
from scipy.integrate import odeint

import matplotlib.pyplot as plt
#plt.rcParams.update({'font.size': 18})

```

We'll define the right-hand side (rhs) of the ordinary differential equations (ODE) using the standard form from the Python basics notebook:

$$\begin{align}
   \frac{d}{dt}\left(\begin{array}{c}
                          \phi \\
                          \dot\phi
                      \end{array}\right)
               = \left(\begin{array}{c}
                          \dot\phi \\
                          -g \sin(\phi)
                       \end{array}\right)
\end{align}$$

```python
def ode_rhs_exact(u_vec, t, *params):
    """
    Right-hand side (rhs) of the differential equation, with
    u_vec = [\phi, \dot\phi] and params = [g, R].  Returns the list of
    d(u_vec)/dt, as prescribed by the differential equation.

    """
    phi, phidot = u_vec  # extract phi and phidot from the passed vector
    g, R = params  # extract g and R from the passed parameters
    return [phidot, -g*np.sin(phi)/R]
```

```python
# parameters
g = 9.8  # in mks units
R = 5    # radius in meters

# absolute and relative tolerances for ode solver
abserr = 1.0e-8
relerr = 1.0e-6

# initial conditions for [phi, phidot]
phi0 = np.pi/180 * 90.  # convert initial phi to radians
u0_vec = [phi0, 0.]

t_max = 15.  # integration time
t_pts = np.arange(0, t_max, 0.01)  # array of time points, spaced 0.01

# Integrate the differential equation and read off phi, phidot (note T!)
phi, phidot = odeint(ode_rhs_exact, u0_vec, t_pts, args=(g, R),
                     atol=abserr, rtol=relerr).T
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(t_pts, 180./np.pi * phi)
fig.tight_layout()  # make the spacing of subplots nicer
```

**Does the plot make sense for $\phi$?  E.g., does it start at the correct angle? Does it have the behavior you expect (e.g., periodic with constant amplitude)?**

Now let's put this into a function:

```python
def solve_for_phi(phi0, phidot0=0, t_min=0., t_max=1., g=9.8, R=5.):
    """
    Solve the differential equation for the skateboard Example 1.2 in Taylor.
    The result for t, \phi(t) and \dot\phi(t) are returned for a grid with
    t_min < t < t_max and a hardwired (for now) spacing of 0.01 seconds.
    The ODE solver is odeint from scipy, with specified tolerances.
    Units are mks and angles are in radians.
    """

    # absolute and relative tolerances for ode solver
    abserr = 1.0e-8
    relerr = 1.0e-6

    # initial conditions for [phi, phidot]
    u0_vec = [phi0, phidot0]

    t_pts = np.arange(t_min, t_max, 0.01)

    # Integrate the differential equation
    phi, phidot = odeint(ode_rhs_exact, u0_vec, t_pts, args=(g, R),
                     atol=abserr, rtol=relerr).T

    return t_pts, phi, phidot
```

Check that it works (gives the previous result).

```python
phi0 = np.pi/180 * 90.  # convert initial phi to radians
t_pts, phi, phidot = solve_for_phi(phi0, t_max=15.)
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(t_pts, 180./np.pi * phi)
fig.tight_layout()  # make the spacing of subplots nicer

```

Ok, now we need an ode function for the small angle approximation.  It's very easy now to copy and modify our other function!

```python
def ode_rhs_small_angle(u_vec, t, *params):
    """
    Right-hand side (rhs) of the differential equation, with
    u_vec = [\phi, \dot\phi] and params = [g, R].  Returns the list of
    d(u_vec)/dt, as prescribed by the differential equation.

    """
    phi, phidot = u_vec  # We don't actually use x or y here, but could!
    g, R = params
    return [phidot, -g*phi/R]
```

And we can put them together into one solver function:

```python
def solve_for_phi_all(phi0, phidot0=0, t_min=0., t_max=1., g=9.8, R=5.):
    """
    Solve the differential equation for the skateboard Example 1.2 in Taylor
    using the exact equation and the small angle approximation.
    The result for t, \phi(t) and \dot\phi(t) are returned for a grid with
    t_min < t < t_max and a hardwired (for now) spacing of 0.01 seconds.
    The ODE solver is odeint from scipy, with specified tolerances.
    Units are mks and angles are in radians.
    """

    # absolute and relative tolerances for ode solver
    abserr = 1.0e-8
    relerr = 1.0e-6

    # initial conditions for [phi, phidot]
    u0_vec = [phi0, phidot0]

    t_pts = np.arange(t_min, t_max, 0.01)

    # Integrate the differential equations
    phi, phidot = odeint(ode_rhs_exact, u0_vec, t_pts, args=(g, R),
                     atol=abserr, rtol=relerr).T
    phi_sa, phidot_sa = odeint(ode_rhs_small_angle, u0_vec, t_pts, args=(g, R),
                     atol=abserr, rtol=relerr).T

    return t_pts, phi, phidot, phi_sa, phidot_sa
```

Always try it out!

```python
phi0 = np.pi/180 * 90.
t_pts, phi, phidot, phi_sa, phidot_sa = solve_for_phi_all(phi0, t_max=15.)

```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(t_pts, 180./np.pi * phi)
ax.plot(t_pts, 180./np.pi * phi_sa)
fig.tight_layout()  # make the spacing of subplots nicer

```

This is actually the plot that is requested, so we could analyze it at this stage, but instead let's improve the plot and see how to save it.

### Ok, now for some more systematic plotting

Here we see examples of applying limits to the x and y axes as well as labels and a title.

```python
fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(1,1,1)
ax.set_xlim(0.,15.)
ax.set_ylim(-25.,25.)
ax.set_xlabel('t (sec)')
ax.set_ylabel(r'$\phi$')
ax.set_title(r'$\phi_0 = 90$ degrees')
line_exact, = ax.plot(t_pts, 180./np.pi * phi, label='exact')
line_sa, = ax.plot(t_pts, 180./np.pi * phi_sa, label='small angle')
ax.legend()

# save the figure
fig.savefig('Taylor_prob_1.51.png', bbox_inches='tight')
```

### Bonus: repeat with widgets!

This actually generalizes problems 1.50 and 1.51 so that you can examine any angle in between.  Use it to check your figure for 1.51.

```python
from ipywidgets import interact, fixed
import ipywidgets as widgets

def rad_to_deg(theta_rad):
    """Take as input an angle in radians and return it in degrees."""
    return 180./np.pi * theta_rad

def deg_to_rad(theta_deg):
    """Take as input an angle in degrees and return it in radians."""
    return np.pi/180. * theta_deg

```

```python
def plot_exact_and_small_angle(phi0_deg=0):
    phi0_rad = deg_to_rad(phi0_deg)
    t_pts, phi_rad, phidot, phi_sa_rad, phidot_sa = \
         solve_for_phi_all(phi0_rad, t_max=15.)
    phi_deg = rad_to_deg(phi_rad)
    phi_sa_deg = rad_to_deg(phi_sa_rad)

    fig = plt.figure(figsize=(8,6))
    ax = fig.add_subplot(1,1,1)
    line_exact, = ax.plot(t_pts, phi_deg, label='exact')
    line_sa, = ax.plot(t_pts, phi_sa_deg, label='small angle')
    ax.legend()
    ax.set_xlim(0.,15.)
    #ax.set_ylim(-90.,90.)
    ax.set_xlabel('t (sec)')
    ax.set_ylabel(r'$\phi$')
    ax.set_title(fr'$\phi_0 = {phi0_deg:.0f}$')
    plt.show()

```

```python
interact(plot_exact_and_small_angle, phi0_deg=(0.,90.));
```

    interactive(children=(FloatSlider(value=0.0, description='phi0_deg', max=90.0), Output()), _dom_classes=('widg…

```python
# to avoid the jiggling and do some formatting
phi0_deg_widget = widgets.FloatSlider(min=0., max=120.0, step=0.1, value=0.,
                                     description=r'$\phi_0$ (degrees)',
                                     readout_format='.0f',
                                     continuous_update=False
                                    )
interact(plot_exact_and_small_angle, phi0_deg=phi0_deg_widget);
```

```python

```


# Exercice 2

```python
%matplotlib inline
from sympy.interactive import printing
printing.init_printing()
from frame import *
import sympy as sp
import numpy as np
```

Nous allons initialiser les differentes valeurs :

```python
E=1.3 #en MPa
h=7.5 #en mm
b=20. #en mm
Lx=55. #en mm
Lyh=60. #en mm
Lyb=45 #en mm
I=b*(h**3)/12 #en mm^4
S=b*h  #en mm^2
eps=10**(-3)
g=9.81
```

Nous allons maintenant créer les noeuds et les éléments de la structure :

```python
nodes= np.array([[0.,0.],[0.,Lyb],[0.,Lyh+Lyb],[Lx/2,Lyh+Lyb],[Lx,Lyh+Lyb],[Lx,Lyb],[Lx,0.]])
elements=np.array([[0,1],[1,5],[1,2],[2,3],[3,4],[4,5],[5,6]])

frame=LinearFrame(nodes,elements)
frame.plot_with_label()
```

```python
ne = frame.nelements
ndof = frame.ndof
EI = np.ones(ne)*E*I
ES = np.ones(ne)*E*S
f_x = 0*np.ones(7)
f_y = 0*np.ones(7)
frame.set_distributed_loads(f_x, f_y)
frame.set_stiffness(EI, ES)
blocked_dof = np.array([0, 1, 2, ndof-3, ndof-2, ndof-1])
bc_values = np.array([0, 0, 0, 0, 0, 0])
```

```python
K = frame.assemble_K()

```

```python
u=np.array([0.,0.,0.,
            2.,0.,0.,
            5.,0.,0.,
            0.,-3.,0.,
            -3.,0.,0.,
            -2.,0.,0.,
            0.,0.,0.])

print (u)

```

    [ 0.  0.  0.  2.  0.  0.  5.  0.  0.  0. -3.  0. -3.  0.  0. -2.  0.  0.
      0.  0.  0.]

```python
F= np.dot(K,np.transpose(u))
F

```

    array([ -2.40740741e-01,  -5.15939161e-16,   5.41666667e+00,
             1.42702152e+01,  -7.17477895e-17,   9.98697917e+00,
             3.56068892e+01,   1.58226897e+00,   2.63265108e+01,
            -1.41818182e+01,  -3.16453794e+00,   0.00000000e+00,
            -2.13235085e+01,   1.58226897e+00,  -2.32796358e+01,
            -1.43717777e+01,   3.20043511e-16,  -6.94010417e+00,
             2.40740741e-01,  -5.15939161e-16,  -5.41666667e+00])

```python
m=F[10]/g
m
```

La masse accrochée serait d'environ 320g

# Exercice 3

Dans cette exercice, on doit créer des fonctions à l'intérieur de notre classe frame afin de récupérer l'effort normal(N), l'effort tangentielle (T) et le moment sur z (M).

```python
def find_N(self,element):
        """
        Returns the normal force of an element.
        """
        F = self.assemble_F()
        N = F[3*element]
        return N

def find_T(self,element):
        """
        Returns the tangential force of an element.
        """
        F = self.assemble_F()
        T = F[3*element+1]
        return T

def find_M(self,element):
        """
        Returns the moment of an element.
        """
        F = self.assemble_F()
        M = F[3*element+2]
        return M
```


# Linear and Quadratic Discriminant Analysis

## Linear Discriminant Analysis

### Classifying with Bayes' Theorem

In a previous chapter we discussed logistic regression for the case of two response classes (e.g. 0 and 1). It models the conditional probability $\Pr(Y=k|X=x)$ directly through the use of the Sigmoid function. In this chapter we discuss an alternative approach that models the distribution of the predictors $X$ separately for each response class (i.e. given $Y$), and then uses Bayes' theorem to transform these into conditional probabilities for $\Pr(Y=k|X=x)$. Its main advantage compared to logistic regressions is that if classes are well-separated, parameter estimates for logistic regression tend to be unstable whereas linear discriminant analysis (LDA) does not suffer from this problem. Beyond that, LDA is a popular algorithm for multiple-class classification (i.e. the response has more than two classes, for example buy/hold/sell etc.) where logistic regression is not used that often (James et al. (2013)).

LDA assigns an object to class $k$ for which the computed probability is highest. These probabilities are calculated using Bayes' rule which states that

\begin{equation}
\begin{aligned}
\underbrace{\Pr(Y=k|X)}_{\text{posterior probability}} &= \frac{\overbrace{\Pr(X|Y=k)}^{\text{conditional probability}}  \quad \overbrace{\Pr(k)}^{\text{prior probability}}}{\underbrace{\Pr(X)}_{\text{evidence}}} \\[3ex]
 &= \frac{\Pr(X|Y=k) \Pr(k)}{\sum_{\ell=1}^K \Pr(X|Y=\ell) \Pr(\ell)}
\end{aligned}
\end{equation}

Above, $\Pr(k)$ is simply the prior probability of class $k$ (with $\sum_{k=1}^K \Pr(k) = 1$) that a randomly chosen observation is drawn from the $k$th class. $\Pr(X|Y=k)$ on the other hand is the class conditional density of $X$ in class $Y=k$. Following the notation in (Friedman et al. (2001)), we denote $\Pr(X|Y=k) \equiv f_k(x)$ to indicate that it is a density function. LDA's decision rule thus classifies an observation into class $k$ if

\begin{align}
\Pr(Y=k|X) &> \Pr(Y=j|X) \qquad \forall j \neq k \nonumber \\
\end{align}

or, if we substitute both sides of the inequality with Bayes' rule:

\begin{align}
\frac{f_k(x) \Pr(k)}{\sum_{\ell=1}^K f_\ell(x) \Pr(\ell)} &> \frac{f_j(x) \Pr(j)}{\sum_{\ell=1}^K f_\ell(x) \Pr(\ell)} \qquad \forall j \neq k \nonumber
\end{align}

The evidence term (denominator in above equation) can be omitted from the decision rule because it is merely a scaling factor (Raschka (2014)). This then yields the following simple decision boundary:

\begin{align}
f_k(x) \Pr(k) &> f_j(x) \Pr(j)  \qquad \forall j \neq k
\end{align}

This expression can also be written as
\begin{equation}
\delta_k(x) = \arg \max_k \; f_k(x) \Pr(k)
\end{equation}

### Bayes Decision Rule in LDA with One Feature

Suppose that $f_k(x)$ follows a Gaussian distribution. For the one-dimensional setting, that is if we have just one feature $p=1$, the normal density takes the well known form

\begin{equation}
f_k(x) = \frac{1}{\sqrt{2\pi \sigma_k^2}} \, \exp\left( - \frac{(x - \mu_k)^2}{2 \sigma_k^2} \right)
\end{equation}

where $\mu_k$ and $\sigma_k^2$ are the mean and variance for the $k$th class, respectively. For the moment let us also assume that there is a shared variance term across all $K$ classes, i.e. $\sigma_1^2 = \sigma_2^2 = \ldots = \sigma_k^2$. Then plugging the normal distribution into our maximization problem, taking the log and doing some algebra - see the appendix of the script for detailed steps - we find that an observation is assigned to class $k$ for which $\delta_k(x)$ is greatest:

\begin{equation}
\delta_k(x) = \arg \max_k \left[\frac{x \mu_k}{\sigma^2} - \frac{\mu_k}{2\sigma^2} + \ln(\Pr(k)) \right]
\end{equation}

Below figure shows how LDA classifies data based on the above result. In the left subplot we see two separate normal densities representing a situation with two classes ($K \in \{\text{blue, green\}}$). $\Pr(k=\text{blue}) = \Pr(k=\text{green}) = 0.5$; equal for both classes. Both densities have the same variance $\sigma_1^2 = \sigma_2^2 = 1$ but different location parameter, $\mu_1 = -1.25, \mu_2 = 1.25$.

If we were to know these parameter, then LDA's decision boundary would be drawn exactly at zero (dashed line). If $\Pr(k=\text{blue}) > \Pr(k=\text{green})$, Bayes' decision boundary would move to the right, if $\Pr(k=\text{blue}) < \Pr(k=\text{green})$, to the left. There is some overlapping area leading to some uncertainty, but overall the error rate is minimized to a minimum. In reality however, we do not know the true location and scale parameter and hence we have to estimate them - what we will discuss momentarily. The right plot displays histograms of 50 randomly drawn observations from the aforementioned normal distribution. Given this data, LDA calculates $\mu_k, \sigma^2$ and uses $\delta_k(x)$ to draw the decision boundary (solid vertical line). Data points to the left belong to the blue class, all others to the green class. The dashed vertical line again displays the optimal decision boundary. Because we don't know the true location and scale parameter LDA relies on estimates. This introduces inaccuracy that is reduced the larger the data sample is (assuming our normal assumption is correct).

### Assumptions and Parameter Estimation

So far we have discussed how LDA draws its decision boundary with the help of Bayes rule and given the assumptions that the features follow a normal distribution. But in order to follow through with our classification task, estimates for $\Pr(k), \mu_k$, and $\sigma^2$ are required. Estimating the prior probability $\Pr(k)$ is no difficult job: we simply compute the fraction of training observations that belong to the $k$th class: $\hat{\Pr}(k) = n_k / n$, where $n_k$ represents the count of samples from class $k$ and $n$ the count of all samples. Location parameter $\mu_k$ is estimated using the average of training observation of the $k$th class and $\sigma^2$, the scale parameter, is the weighted average of the sample variance for each of the $K$ classes (Note that Friedman et al. (2001) and James et al (2013) both use a biased corrected version of $\hat{\sigma}^2$ (and $\hat{\Sigma}$ for the case of $p>1$) by dividing the summed terms by $n-K$ instead of $n$. The formula given here uses an uncorrected estimate of $\sigma$ and in that follows `Sklearn`'s implementation.)

\begin{align}
\hat{\mu}_k &= \frac{1}{n_k} \sum_{i:y_i=k} x_i \\
\hat{\sigma}^2 &= \frac{1}{n} \sum_{k=1}^K \sum_{i:y_i=k} (x_i - \hat{\mu}_k)^2
\end{align}

Given the assumption of normality and given these estimates for location and scale we are able to establish a decision rule that assigns each new data point to the class for which $\delta_k(x)$ is highest.

### Bayes Decision Rule in LDA with More Than One Feature

Above we have used the one-dimensional case with one predictor to introduce how LDA classifies an observation. Now we extend the classifier to work with multiple features ($p>1$). Again we assume that $X = (X_1, X_2, \ldots, X_p)$ is drawn from a (multivariate) normal distribution with a class specific mean vector $\mu_k$ of length $p$ and common covariance matrix $\Sigma$ of dimension $p \times p$. This is expressed as $X \sim N(\mu, \Sigma)$. The multivariate Gaussian density is defined as

\begin{equation}
f_k(x) = \frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}} \, \exp \left( -\frac{1}{2} (x-\mu_k)^T \Sigma^{-1} (x-\mu_k) \right)
\end{equation}

As before we plug this expression into our maximization problem, take the logarithm and perform a little bit of algebra (For the interested the different steps are shown in the appendix of the script). This yields to the following LDA's Bayes classifier rule, based on which an observation $X=x$ is assigned to the class for which $\delta_k(x)$ is largest.

\begin{equation}
\delta_k(x) = \arg \max_k \left[x^T \Sigma^{-1} \mu_k - \frac{1}{2} \mu_k^T\Sigma^{-1}\mu_k + \ln(\Pr(k))\right]
\end{equation}

The estimates for $\Pr(k), \mu_k$, and $\Sigma$ follow again the same approach as in the case of only one predictor.

The next figure plots LDA's Bayes decision boundary for a random training set with two features $X_1, X_2$. The colors indicate the binary response with blue circles indicating customers who accepted a product offer and green circles representing those who declined it. The bivariate normal contours (ellipses) represent iso-lines with the same probabilities. LDA uses Bayes' decision rule discussed above to classify any new data point into class $k$.

### LDA in Python
#### Setup

We will apply LDA in Python with the functions that are provided through the `sklearn` package and the 'Default' data set we used to introduce logistic regression in a previous chapter. `Sklearn`, short for Scikit-learn, is a key resource for clustering, classification or regression algorithms in machine learning. It offers an abundant variety of functions and functionalities and is actively developed by a large community.

`Sklearn` is one of the most extensive package in Python with hundreds, if not thousands, of functions. It is good practice to not load the full library as we did for example with `numpy` but to only load those functions that are needed to run your task at hand. This saves computer memory and with that improves the efficiency of your algorithm, especially if you are using your household PC to run it on larger data sets.

```python
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn import metrics
plt.rcParams['font.size'] = 14
plt.style.use('seaborn-whitegrid')
```

```python
# Default data set is not available online. Data was extracted from R package "ISLR"
df = pd.read_csv('Data/Default.csv', sep=',')

# Factorize 'No' and 'Yes' in columns 'default' and 'student'
df['defaultFac'] = df.default.factorize()[0]
df['studentFac'] = df.student.factorize()[0]
df.head(5)
```

      default
      student
      balance
      income
      defaultFac
      studentFac

      0
      No
      No
      729.526495
      44361.625074
      0
      0

      1
      No
      Yes
      817.180407
      12106.134700
      0
      1

      2
      No
      No
      1073.549164
      31767.138947
      0
      0

      3
      No
      No
      529.250605
      35704.493935
      0
      0

      4
      No
      No
      785.655883
      38463.495879
      0
      0

```python
# Assign data to feature matrix X_train and response vector y_train
X_train = df[['balance', 'income', 'studentFac']]
y_train = df.defaultFac
```

#### LDA Classifier Object & Fit
Now we are in a position to run the LDA classifier. This, as you can see from the three lines below, is as easy as it gets.

```python
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

# Create LDA object and run classifier
lda = LDA(solver='lsqr')
lda = lda.fit(X_train, y_train)
lda
```

    LinearDiscriminantAnalysis(n_components=None, priors=None, shrinkage=None,
                               solver='lsqr', store_covariance=False, tol=0.0001)

The parameter `solver='lsqr'` specifies the method by which the covariance matrix is estimated. `lsqr` follows the approach introduced in the preceding subsection. Others such as `svd` or `eigen` are available. See [Scikit-learn's guide](http://scikit-learn.org/stable/modules/lda_qda.html#estimation-algorithms) or the [function description](http://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html).

Every function in `sklearn` has different attributes and methods. `Sklearn`s convention is to store anything that is derived from the data in attributes that end with a trailing underscore. That is to separate them from parameters that are set by the user (Mueller and Guido (2017)). For example the estimated covariance matrix can be printed with this command.

```python
lda.covariance_
```

    array([[ 2.05277550e+05, -9.37165654e+05,  4.21453998e+01],
           [-9.37165654e+05,  1.77777941e+08, -4.57857337e+03],
           [ 4.21453998e+01, -4.57857337e+03,  2.07468022e-01]])

In a Jupyter notebook, to see all options you can simply type `lda.` and hit tab.

#### LDA Performance

Here are some basic metrics on how the LDA classifier performed on the training data.

```python
print('default-rate: {0: .4f}'.format(np.sum(y_train)/len(y_train)))
print('score:        {0: .4f}'.format(lda.score(X_train, y_train)))
print('error-rate:   {0: .4f}'.format(1-lda.score(X_train, y_train)))
```

    default-rate:  0.0333
    score:         0.9725
    error-rate:    0.0275

Overall, 3.33% of all observations defaulted. If we would simply label each entry as 'non-default' we would have an error rate of this magnitude. So, in comparison to this *naive* classifier, LDA seems to have some skill in predicting the default.

> **IMPORTANT NOTE: In order to be in line with James et al. (2015), the textbook for this course, we have not performed any train/test split of the data. Therefore we will use the same full matrix `X_train` and response vector `y_train` as test data. Performance metrics might be applied to both test and training data but in the end the results on the test set are those that we are ultimately interested in. To drive this point home, I have relabeled the  `X_train` and `y_train` to `X_test`, `y_test`.  Nevertheless, be aware that in this unique case it is the same data!**

Let us print the confusion matrix introduced in the previous chapter to see the class-wise performance. For reference the confusion matrix is also printed as `DataFrame` but moving forward be sure to know that row represent the true values and columns predicted labels.

```python
# Relabel variables as discussed
X_test = X_train
y_test = y_train

# Predict labels
y_pred = lda.predict(X_test)

# Sklearn's confusion matrix
print(metrics.confusion_matrix(y_test, y_pred))

# Manual confusion matrix as pandas DataFrame
confm = pd.DataFrame({'Predicted default status': y_pred,
                      'True default status': y_test})
confm.replace(to_replace={0:'No', 1:'Yes'}, inplace=True)
print(confm.groupby(['True default status','Predicted default status']).size().unstack('Predicted default status'))
```

    [[9645   22]
     [ 253   80]]
    Predicted default status    No  Yes
    True default status
    No                        9645   22
    Yes                        253   80

The confusion matrix tells us that for the non-defaulters, LDA only misclassified 22 of them. This is an excellent rate. However, out of the 333 (=253 + 80) people who actually defaulted, LDA classified only 80 correctly. This means our classifier missed out on 76.0% of those who actually defaulted! For a credit card applicant with a bad credit score this is good news. For a credit card company, not so much.

#### Varying the Threshold Levels
Why does LDA miss all these 'defaulters'? Implicitly, Bayes classifier minimizes the **overall** error rate, meaning that it yields the smallest possible total number of misclassified observations - irrespective of the class-specific error rate. Bayes classifier works by assigning an observation to class 'default' for which the posterior probability $Pr(\text{default = Yes}|X=x) > 0.5$. For a credit card company who seeks to have as few defaults as possible, this threshold might be too large. Instead, such a company might decide to label any customer with a posterior probability of default above 20% to the 'default' class ($Pr(\text{default = Yes}|X=x) > 0.2$). Let us investigate how the results in such a case would look like.

```python
# Calculated posterior probabilities
posteriors = lda.predict_proba(X_test)
posteriors[:5, :]
```

    array([[0.996778  , 0.003222  ],
           [0.99731184, 0.00268816],
           [0.98529382, 0.01470618],
           [0.99881647, 0.00118353],
           [0.99597848, 0.00402152]])

The function `lda.predict_proba()` provides the posterior probabilities of $\Pr(\text{default = 0}|X=x)$ in the first column and $\Pr(\text{default = 1}|X=x)$ in the second. The latter column is what we are interested in. Out of convenience we use `sklearn`'s `binarize` function to classify all probabilities above the threshold of 0.2 as 1 (=default) and generate the confusion matrix.

```python
from sklearn.preprocessing import binarize

# Set threshold and get classes
thresh = 0.2
y_pred020 = binarize([posteriors[:, 1]], thresh)[0]

# new confusion matrix (threshold of 0.2)
print(metrics.confusion_matrix(y_test, y_pred020))
```

    [[9435  232]
     [ 140  193]]

Now LDA misclassifies only 140 out of 333 defaults, or 42.0%. Thats a sharp improvement over the 76.0% from before. But this comes at a price: Before, of those who did not default LDA mislabeled only 22 (or 0.2%) incorrectly. This number increased now to 232 (or 2.4%). Combined, the total error rate increased from 2.75% to 3.72%. For a credit card company, this might be a price they are willing to pay to have a more accurate identification of individuals who default.

Below code snippet calculates and plots the overall error rate, the proportion of missed defaulting customers and the fraction of error among the non-defaulting customers as a function of the threshold value for the posterior probability that is used to assign classes.

```python
# Array of thresholds
thresh = np.linspace(0, 0.5, num=100)

er   = []  # Total error rate
der  = []  # Defaults error rate
nder = []  # Non-Defaults error rate

for t in thresh:
    # Sort/arrange data
    y_pred_class = binarize([posteriors[:, 1]], t)[0]
    confm = metrics.confusion_matrix(y_test, y_pred_class)

    # Calculate error rates
    er   = np.append(er, (confm[0, 1] + confm[1, 0]) / len(posteriors))
    der  = np.append(der, confm[1, 0] / np.sum(confm[1, :]))
    nder = np.append(nder, confm[0, 1] / np.sum(confm[0, :]))
```

```python
# Plot
plt.figure(figsize=(12, 6))
plt.plot(thresh, er, label='Total error rate')
plt.plot(thresh, der, label='Missed defaults')
plt.plot(thresh, nder, label='Missed non-defaults')
plt.xlim(0, 0.5)
plt.xlabel('Threshold')
plt.ylabel('Error Rate')
plt.legend();
```

How do we know what threshold value is best? Unfortunately there's no formula for it. "Such a decision must be based on *domain knowledge*, such as detailed information about costs associated with defaults" (James et al. (2013, p.147)) and it will always be a **trade-off: if we increase the threshold we reduce the missed non-defaults but at the same time increase the missed defaults**.

## Performance Metrics

This is now the perfect opportunity to refresh our memory on a few classification performance measures introduced in the previous chapters and add a few more to have a full bag of performance metrics. The following table will help in doing this.

We will use the following abbreviations ([Markham (2016)](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb)):

* True Positives (TP): correctly predicted defaults
* True Negatives (TN): correctly predicted non-defaults
* False Positives (FP): incorrectly predicted defaults ("Type I error")
* False Negatives (FN): incorrectly predicted non-defaults ("Type II error")

```python
# Assign confusion matrix values to variables
confm = metrics.confusion_matrix(y_test, y_pred)
print(confm)
TP = confm[1, 1]  # True positives
TN = confm[0, 0]  # True negatives
FP = confm[0, 1]  # False positives
FN = confm[1, 0]  # False negatives
```

    [[9645   22]
     [ 253   80]]

So far we've encountered the following performance metrics:

* Score
* Error rate
* Sensitivity and
* Specificity.

We briefly recapture their meaning, how they are calculated and how to call them in Scikit-learn. We will make use of the functions in the `metrics` sublibrary of `sklearn`

#### Score
* Score = (TN + TP) / (TN + TP + FP + FN)
* Fraction of (overall) correctly predicted classes

```python
print((TN + TP) / (TN + TP + FP + FN))
print(metrics.accuracy_score(y_test, y_pred))
print(lda.score(X_test, y_test))
```

    0.9725
    0.9725
    0.9725

#### Error rate
* Error rate = 1 - Score or Error rate = (FP + FN) / (TN + TP + FP + FN)
* Fraction of (overall) incorrectly predicted classes
* Also known as "Misclassification Rate"

```python
print((FP + FN) / (TN + TP + FP + FN))
print(1 - metrics.accuracy_score(y_test, y_pred))
print(1 - lda.score(X_test, y_test))
```

    0.0275
    0.02749999999999997
    0.02749999999999997

#### Specificity

* Specificity = TN / (TN + FP)
* Fraction of correctly predicted negatives (e.g. 'non-defaults')

```python
print(TN / (TN + FP))
```

    0.9977242164063308

#### Sensitivity or Recall
* Sensitivity = TP / (TP + FN)
* Fraction of correctly predicted 'positives' (e.g. 'defaults'). Basically asks the question: "When the actual value is positive, how often is the prediction correct?"
* Also known as **True positive rate**
* Counterpart to *Precision*

```python
print(TP / (TP + FN))
print(metrics.recall_score(y_test, y_pred))
```

    0.24024024024024024
    0.24024024024024024

The above four classification performance metrics we already encountered. There are two more metrics we want to cover: **Precision** and the **F-Score**.

#### Precision
* Precision = TP / (TP + FP)
* Refers to the accuracy of a positive ('default') prediction. Basically asks the question: "When a positive value is predicted, how often is the prediction correct?"
* Counterpart to *Recall*

```python
print(TP / (TP + FP))
print(metrics.precision_score(y_test, y_pred))
```

    0.7843137254901961
    0.7843137254901961

#### F-Score

Van Rijsbergen (1979) introduced a measure that is still widely used to evaluate the accuracy of predictions in two-class (binary) classification problems: the F-Score. It combines Precision and Recall (aka Sensitivity) in one metric and tells us something about the relations between data's positive labels and those given by a classifier. It is a single measure of a classification procedure's usefullness and in general the rule is that the higher the F-Score, the better the predictive power of the classification procedure. It is defined as:

\begin{align}
F_{\beta} &= \frac{(1 + \beta^2) \cdot \text{precision} \cdot \text{recall}}{\beta^2 \cdot \text{precision} + \text{recall}} \\[2ex]
          &= \frac{(1+\beta^2) \cdot TP}{(1+\beta^2) \cdot TP + \beta^2 \cdot FN + FP}
\end{align}

This measure employs a parameter $\beta$ that captures a user's preference (Guggenbuehler (2015)). The most common value for $\beta$ is 1. This $F_1$-score weights both precision and recall evenly (simple harmonic mean). In rare cases the $F_2$-score is used, which puts twice as much weights on recall as on precision (Hripcsak and Rotschild (2005)).

```python
print(metrics.confusion_matrix(y_test, y_pred))
print(metrics.f1_score(y_test, y_pred))
print(((1+1**2) * TP)/((1+1**2) * TP + FN + FP))
print(metrics.classification_report(y_test, y_pred))
```

    [[9645   22]
     [ 253   80]]
    0.367816091954023
    0.367816091954023
                  precision    recall  f1-score   support

               0       0.97      1.00      0.99      9667
               1       0.78      0.24      0.37       333

        accuracy                           0.97     10000
       macro avg       0.88      0.62      0.68     10000
    weighted avg       0.97      0.97      0.97     10000

Let us compare this to the situation where we set the posterior probability threshold for 'default' at 20%.

```python
# Confusion matrix & clf-report for cut-off
# value Pr(default=yes | X = x) > 0.20
print(metrics.confusion_matrix(y_test, y_pred020))
print(metrics.classification_report(y_test, y_pred020))
```

    [[9435  232]
     [ 140  193]]
                  precision    recall  f1-score   support

               0       0.99      0.98      0.98      9667
               1       0.45      0.58      0.51       333

        accuracy                           0.96     10000
       macro avg       0.72      0.78      0.74     10000
    weighted avg       0.97      0.96      0.96     10000

We see that by reducing the cut-off level from $\Pr(\text{default} = 1| X=x) > 0.5$ to $\Pr(\text{default} = 1| X=x) > 0.2$ precision decreases but recall improves. This changes the $F_1$-score.

Does this mean that a threshold of 20% is more appropriate? In general, one could argue for a 'yes'. Yet, as mentioned before, this boils down to domain knowledge. Where the $F_1$-score is of help, together with the other metrics introduced, is when we compare models against each other and want to determine which one performed best. For example if we compare results from logistic regression and LDA (and both used a cut-off level of 50%) the F-score would suggest that the one with the higher value performed better.

For a summary on performance metrics the following two ressources are recommended:

* For the interested reader an excellent and easily accessible summary on performance metrics is the article by [Sokolova and Lapalme (2009)](http://rali.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf).

* For further details and examples please also consider the [scikit-learn discription](https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-f-measure-metrics).

## Precision-Recall Curve

If one is interested in understanding how precision and recall varies given different level of thresholds, then there is a function to do this.

```python
# Extract data displayed in above plot
precision, recall, threshold = metrics.precision_recall_curve(y_test, posteriors[:, 1])

print('Precision: ', precision)
print('Recall:    ', recall)
print('Threshold: ', threshold)
```

    Precision:  [0.05540765 0.05525046 0.05525965 ... 1.         1.         1.        ]
    Recall:     [1.         0.996997   0.996997   ... 0.00600601 0.003003   0.        ]
    Threshold:  [0.00227206 0.00227354 0.00227411 ... 0.92183985 0.93353321 0.94256071]

This one can easily visualize - done in the next code snippet. We also add some more information to the plot by displaying the *Average Precision (AP)* and the *Area under the Curve (AUC)*. The former summarizes the plot in that it calculates the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight [see further description here](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score). The latter calculates the area under the curve using the trapezoidal rule. Notice that ideally the function hugs the top-right corner.

```python
# Calculate the average precisions score
y_dec_bry = lda.decision_function(X_test)
average_precision = metrics.average_precision_score(y_test, y_dec_bry)

# Calculate AUC
prec_recall_auc = metrics.auc(recall, precision)

# Plot Precision/Recall variations given different
# levels of thresholds
plt.plot(recall, precision)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('2-class Precision-Recall curve: \n AP={0:0.2f} / AUC={1:0.2f}'.format(
          average_precision, prec_recall_auc));
```

## ROC Curve

Having introduced the major performance measures, let us now discuss the so called ROC curve (short for "receiver operating characteristics"). This is a very popular way of visualizing the performance of binary classifiers. Its origin are in signal detection theory durign WWII (Flaach (2017)) but it has since found application in medical decision making and machine learning ([Fawcett (2006)](http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf)). ROC investigates the relationship between sensitivity and specificity of a binary classifier. Sensitivity (or true positive rate) measures the proportion of positives (defaults) correctly classified. Specificity (or true negative rate) measures the proportion of negatives (non-defaults) correctly classified.

Above we calculated that if we use $\Pr(\text{default = Yes}|X=x) > 0.5$ to classify posterior probabilities as defaults, LDA has its best overall error rate but misses on 76.0% of the customers who acutally defaulted. By decreasing the threshold to 0.2 we improved the accuracy of detecting defaults but this came at the cost of a higher overall error rate. This was the trade-off we faced. The ROC curve serves to visualize a variation of this trade-off. It varies the cut-off threshold from 0 to 1 and calculates for each threshold the true positive rate (aka sensitivity) and false positive rate (equals 1 - specificity). These values are then plotted with the former on the vertical and the later on the horizontal axis.

Though this might feel a bit abstract if one is not familiar with all these technical terms, the interpretation is fortunately fairly simple. The ideal ROC curve will hug the top left corner. In that case, the area under the curve (AUC) is biggest. The bigger the AUC, the better the classifier. A perfect classifier has an AUC of 1.

Here's how we calculate the ROC numbers, the corresponding area under the curve and how we plot it.

```python
# Compute ROC curve and ROC area (AUC) for each class
fpr, tpr, thresholds = metrics.roc_curve(y_test, posteriors[:, 1])
roc_auc = metrics.auc(fpr, tpr)
```

```python
plt.figure(figsize=(6, 6))
plt.plot(fpr, tpr, lw=2, label='ROC curve (area = {0: 0.2f})'.format(roc_auc))
plt.plot([0, 1], [0, 1], lw=2, c = 'k', linestyle='--')
plt.xlim([-0.01, 1.0])
plt.ylim([-0.01, 1.01])
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.title('Receiver operating characteristic (ROC)', fontweight='bold', fontsize=18)
plt.legend(loc="lower right");
```

An AUC value of 0.95 is close to the maximum of 1 and should be deemed very good. The dashed black line puts this in perspective: it represents the "no information" classifier; this is what we would expect if the probability of default is not associated with 'student' status and 'balance'. Such a classifier, that performs no better than chance, is expected to have an AUC of 0.5.

## Quadratic Discriminant Analysis

### Underlying Assumptions

For LDA we assume that observations within each class are drawn from a multivariate normal distribution with a class-specific mean vector and a common covariance metrix: $X \sim N(\mu_k, \Sigma)$. Quadratic discriminant analysis (QDA) relaxes these assumptions somewhat. The basic assumption is still that the observations follow a multivariate normal distribution, however, QDA allows for class specific means and covariance matrices: $X \sim N(\mu_k, \Sigma_k)$, where $\Sigma_k$ is a covariance matrix for the $k$th class. With that, the Bayes classifier assigns an observation to the class for which

\begin{equation}
\delta_k(x) = \arg \max_k \; - \frac{1}{2} \ln(|\Sigma_k|) - \frac{1}{2} (x - \mu_k)^T \Sigma_k^{-1} (x - \mu_k) + \ln(\Pr(k))
\end{equation}

is highest. For a derivation of this result see again the appendix of the script. As was the case for LDA, parameter $\mu_k, \Sigma_k$ and $\Pr(k)$ are again estimated from the training data with the same formulas introduced in this notebook.

Below figure depict both LDA and QDA. Both classifiers were trained on the same data. Due to the different variability of the two classes the QDA algorithm seems to perform slightly better in this case.

Under what circumstances should we prefer QDA over LDA? As always, there's no straight answer to this question. Obviously, performance should be king. However, it is said that LDA tends to be a better bet than QDA if the training set is small. In contrast, if the hold-out set is large or the assumption of a common covariance matrix is clearly incorrect, then QDA is recommended. Beyond that, we have to keep in mind that QDA estimates $K p(p+1)/2$ parameters. So if the number of parameters $p$ is large, QDA might take some time to process (James et al. (2013)).

> **Naive Bayes**

> Naive Bayes is the name for a family of popular ML algorithms that are often used in text mining. Text mining is a field of ML that deals with extracting quantitative information from text. A simple example of it is the analysis of Twitter feeds in order to predict stock market reactions. There exist different variations of Naive Bayes applications. One is called 'Gaussian Naive Bayes' and works similar to QDA - with the exception that contrary to QDA the covariance matrices $\Sigma$ are assumed to be diagonal. This means $\Sigma_k$ only contains the variances of the different features for class $k$. Its covariance terms (the off-diagonal elements) are assumed to be zero. Because of its popularity, Naive Bayes is well documented in text books and on the web. A good starting point is Scikit-learn's tutorial on [Naive Bayes](http://scikit-learn.org/stable/modules/naive_bayes.html), Collins (2013) or Russell and Norvig (2009, p.499). To apply the algorithm in Python you want to use `sklearn.naive_bayes.GaussianNB()` or (for text mining preferably) `sklearn.naive_bayes.MultinomialNB()`.

### QDA in Python

The application of QDA follows the one detailed for LDA. Therefore we let the code speak for itself.

```python
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA

# Run qda on training data
qda = QDA().fit(X_train, y_train)
qda
```

    QuadraticDiscriminantAnalysis(priors=None, reg_param=0.0,
                                  store_covariance=False, tol=0.0001)

```python
# Predict classes for qda
y_pred_qda = qda.predict(X_test)
posteriors_qda = qda.predict_proba(X_test)[:, 1]

# Print performance metrics
print(metrics.confusion_matrix(y_test, y_pred_qda))
print(qda.score(X_test, y_test))
print(metrics.classification_report(y_test, y_pred_qda))
```

    [[9636   31]
     [ 239   94]]
    0.973
                  precision    recall  f1-score   support

               0       0.98      1.00      0.99      9667
               1       0.75      0.28      0.41       333

        accuracy                           0.97     10000
       macro avg       0.86      0.64      0.70     10000
    weighted avg       0.97      0.97      0.97     10000

The performance seems to be slightly better than with LDA. Let's plot the ROC curve for both LDA and QDA.

```python
# Compute ROC curve and ROC area (AUC) for each class
fpr_qda, tpr_qda, _ = metrics.roc_curve(y_test, posteriors_qda)
roc_auc_qda = metrics.auc(fpr_qda, tpr_qda)
```

```python
plt.figure(figsize=(6, 6))
plt.plot(fpr, tpr, lw=2, label='LDA ROC (AUC = {0: 0.2f})'.format(roc_auc))
plt.plot(fpr_qda, tpr_qda, lw=2, label='QDA ROC (AUC = {0: 0.2f})'.format(roc_auc_qda))
plt.plot([0, 1], [0, 1], lw=2, c = 'k', linestyle='--')
plt.xlim([-0.01, 1.0])
plt.ylim([-0.01, 1.01])
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.title('ROC Curve', fontweight='bold', fontsize=18)
plt.legend(loc="lower right");
```

With respect to Sensitivity (Recall) and Specificity LDA and QDA perform virtually identical. Therefore, one might give the edge here to QDA because of its slighly better Recall and $F_1$-Score.

## Reality and the Gaussian Assumption for LDA & QDA

Despite the rather strict assumptions regarding normal distribution, LDA and QDA perform well on an amazingly large and diverse set of classification tasks. Friedman et al. (2001, p. 111) put it this way:

> "*Both techniques are widely used, and entire books are devoted to LDA. It seems that whatever exotic tools are the rage of the day, we should always have available these two simple tools. The question arises why LDA and QDA have such a good track record. The reason is not likely to be that the data are approximately Gaussian, and in addition for LDA that the covariances are approximately equal. More likely a reason is that the data can only support simple decision boundaries such as linear or quadratic, and the estimates provided via the Gaussian models are stable. This is a bias variance tradeoff - we can put up with the bias of a linear decision boundary because it can be estimated with much lower variance than more exotic alternatives. This argument is less believable for QDA, since it can have many parameters itself - although perhaps fewer than the non-parametric alternatives.*"

Whether LDA or QDA should be applied to categorical/binary features warrants a separate note. It is true that discriminant analysis was designed for continuous features (Ronald A. Fisher (1936)) where the underlying assumption is that the values are normally distributed. However, as above quote shows, studies have proven the robustness of the model even in light of violations of the rather rigid normality assumption. This is not only true for continuous features but also for categorical/binary features. For more details see Huberty et al. (1986). It follows that applying LDA and QDA is possible, though the user should cautiously control the output. We will discuss appropriate cross validation methods to do so in the next chapter.

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Collins, Michael, 2013, The Naive Bayes Model, Maximum-Likelihood Estimation, and the EM Algorithm, Technical report, Columbia University, New York.
* Fawcett, Tom, 2006, An introduction to ROC analysis, *Pattern Recognition Letters* 27, 861–874.
* Fisher, Roland A., 1936, The Use of Multiple Measurements in Taxonomic Problems, *Annals of Human Genetics* 7, 179-188.
* Flach, Peter A., 2017, Roc analysis, in Claude Sammut, and Geoffrey I. Webb, eds., *Encyclopedia of Machine Learning and Data Mining*, 1109–1116 (Springer Science & Business Media, New York, NY).
* Friedman, Jerome, Trevor Hastie, and Robert Tibshirani, 2001, *The Elements of Statistical Learning* (Springer, New York, NY).
* Guggenbuehler, Jan P., 2015, Predicting Net New Money Using Machine Learning Algorithms and Newspaper Articles, Technical report, University of Zurich, Zurich.
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Jobson, J. David, and Bob Korkie, 1980, Estimation for Markowitz Efficient Portfolios, *Journal of the American Statistical Association* 75, 544–554.
* Hripcsak, George, and Adam S Rothschild, 2005, Agreement, the F-measure, and Reliability in Information Retrieval, *Journal of the American Medical Informatics Association* 12, 296–298.
* Huberty, Carl J., Joseph M. Wisenbaker, Jerry D. Smith, and Janet C. Smith, 1986, Using Categorical Variables in Discriminant Analysis, *Multivariate Behavioral Research* 21, 479-496.
* Ledoit, Olivier, and Michael Wolf, 2004, Honey, i shrunk the sample covariance matrix, *The Journal of Portfolio Management* 30, 110–119.
* Müller, Andreas C., and Sarah Guido, 2017, *Introduction to Machine Learning with Python* (O’Reilly Media, Sebastopol, CA).
* Raschka, Sebastian, 2014, Naive Bayes and Text Classification I - Introduction and Theory from website, http://sebastianraschka.com/Articles/2014_naive_bayes_1.html, 08/31/2017
* Russell, Stuart, and Peter Norvig, 2009, *Artificial Intelligence: A Modern Approach* (Prentice Hall Press, Upper Saddle River, NJ).
* Sokolova, Marina, and Guy Lapalme, 2009, A systematic analysis of performance measures for classification tasks, *Information Processing & Management* 45, 427–437.
* Van Rijsbergen, Cornelis Joost, 1979, *Information Retrieval* (Butterworths, London).

# Addendum

## predict, predict_proba, and decision_function

Let us quickly discuss the difference between the
* `classifier.predict()`,
* `classifier.predict_proba()`, and
* `classifier.decision_function()`.

`classifier.predict()` we already know: it simply predicts the label given the traineded classifier and a feature matrix X (preferably a test set).

```python
lda.predict(X_test)[:10]
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int64)

`classifier.predict_proba()` we have also introduced above: it provides probabilities of $\Pr(y = 0|X=x)$ in the first column and $\Pr(y = 1|X=x)$ in the second.

```python
lda.predict_proba(X_test)[:10]
```

    array([[9.96778003e-01, 3.22199657e-03],
           [9.97311835e-01, 2.68816484e-03],
           [9.85293824e-01, 1.47061759e-02],
           [9.98816468e-01, 1.18353192e-03],
           [9.95978476e-01, 4.02152411e-03],
           [9.95793515e-01, 4.20648549e-03],
           [9.95593550e-01, 4.40644998e-03],
           [9.97315195e-01, 2.68480456e-03],
           [9.77081674e-01, 2.29183256e-02],
           [9.99906164e-01, 9.38359460e-05]])

Finally, `classifier.decision_function()` predicts confidence scores given the feature matrix. The confidence scores for a feature matrix is the signed distance of that sample to the hyperplane. What this exaclty means should become more clear once we have discussed the support vector classifier (SVC).

```python
lda.decision_function(X_test)[:10]
```

    array([-5.73452686, -5.91620475, -4.20467236, -6.73806793, -5.51206468,
           -5.46691242, -5.42026972, -5.91745893, -3.75263341, -9.27386872])

## ROC & Precision-Recall Curve in Sklearn Version 0.22.1

Starting with Scikit-learn version 0.22.1 the plotting of the ROC and Precision-Recall Curve was integrated into Scikit-learn and there's now a function available to cut the plotting work a bit short. Below two code snippets that show how to do it.

```python
# Plot Precision-Recall Curve
disp = metrics.plot_precision_recall_curve(lda, X_test, y_test);
```

```python
disp = metrics.plot_roc_curve(lda, X_test, y_test);
```

Notice that you can also overlay ROCs from multiple models. [See this example](https://scikit-learn.org/stable/auto_examples/plot_roc_curve_visualization_api.html)


SymPy

This notebook shows some of SymPy's main features. Consult the official documentation at  http://docs.sympy.org/latest/index.html to obtain the whole module reference.

SymPy is a Computer Algebra System (CAS) for Python. In other words, you can perform symbolic mathematic operations with it and embed these operations into your Python programs. SymPy is part of the SciPy project. Usually, Sympy is bundled with common Python distribution, such as Anaconda (https://www.continuum.io/anaconda). If it is not bundled with your distribution and you use pip, you can install SymPy with the command "pip install sympy".

Most of the shown examples are derived from the free book "Scipy lectures" (http://www.scipy-lectures.org).

Once you have installed SymPy on your device, you can load it into your Python script:

```python
# This notebook was tested with Python 3.5 and SymPy 1.0

# Only methods that are used in this notebook are imported. To import all functions, write "from sympy import *".
# Note that some methods, such as "sin", have homonymous counterparts in other packages, such as NumPy.
# If you use other packages, use "import sympy (as ...)" and "sympy." before every SymPy method to avoid conflicts.
from sympy import * #e.g. Derivative, diff, dsolve, Eq, Function, init_printing, integrate, lambdify, Matrix, sin, solve, sqrt, ...
from sympy.abc import x, y, z # Can be anything from a to z. Represents variables.
from sympy.solvers import linsolve # To solve linear equation systems.
```

1. Symbolic representation with SymPy

SymPy works with symbolic mathematic operations. You can store mathematic formulas in variables without the necessity to compute them directly:

```python
function = sqrt(sin(x)) + 23 # "sqrt" and "sin" were imported from sympy. "x" was imported from sympy.abc.
function
```

    sqrt(sin(x)) + 23

2. Usage in interactive sessions

In interactive sessions, like this Jupyter notebook, you can also print your expressions as more readable LATEX or Matjax expressions (depending on your device) after you initialzed this option with "init_printing()":

(More information at http://docs.sympy.org/latest/tutorial/printing.html)

```python
init_printing()
function
```

```python
oo #Infinite
```

3. Simplify or expand mathematical expressions

```python
expand((x + y)**3)
```

```python
expand(x + y, complex=True)
```

```python
expand(cos(x + y), trig=True)
```

```python
simplify((x + x*y) / x)
```

4. Limits of functions

```python
# Limit of sin(x) as x->0
limit(sin(x)/x, x, 0)
```

```python
# Using oo (which stands for inifinte)
# Limit of x as x->oo
limit(x, x, oo)
```

```python
limit(x**2, x, -oo)
```

5. Differentiation and Integration

You can perform calculus operations on your functions, such as differentiation and integration:

(More information at http://docs.sympy.org/latest/tutorial/calculus.html)

```python
function2 = x**2
# First derivative
function2.diff()
```

```python
# Second derivative using alternative diff()
diff(function2, x, 2)
```

```python
function3 = sin(x) * 2
# Integration without limits (indefinite)
integrate(function3, x)
```

```python
# Integration with limits (definite)
integrate(function3, (x, 0, 2))
```

```python
# Improper integration (using oo as infinite)
integrate(exp(-x**2), (x, -oo, oo))
```

5. Series expansion

You can find about the series module at http://docs.sympy.org/latest/modules/series/index.html

```python
# Get the Taylor series at 0 until x^15...
series(cos(x), x, 0, 15)
```

```python
# ...or at 3 until the error is automatically assumed as small enough
series(cos(x), x, 3)
```

```python
# Get the Fourier series from -Pi to +Pi
s = fourier_series(x**2, (x, -pi, pi))
s
```

```python
# Returns the 4 first elements of the fourier series
s.truncate(n=4)
```

6. Defining and solving equations/inequalities

Aside of defining functions, you can also define liner and non-linear equations (with "Eq()") and solve them:

(More information at http://docs.sympy.org/latest/modules/solvers/solvers.html)

```python
equation = Eq(x**2, 4)
solve(equation)
```

You can access the left hand side and the right hand side of the equation with the following attributes:

```python
equation.lhs
```

```python
equation.rhs
```

SymPy can also solve inequalities,...

```python
inequality = x**2 < 10
solve(inequality)
```

...linear equation systems...

```python
# Solves the following system of two equations:
# x +  y +  z -1 = 0
# x + 2y + 3z -1 = 0
# Alternative method: linsolve([x + y + z - 1, x + 2*y + 3*z - 1], (x, y, z))
linsolve(Matrix(([1, 1, 1, 1], [1, 2, 3, 1])), (x, y, z))
```

...and overdetermined systems

```python
x1 = Symbol('x1')
x2 = Symbol('x2')
f1 = 3 * x1**2 - 2 * x2**2 - 1
f2 = x1**2 - 2 * x1 + x2**2 + 2 * x2 - 8
print(nsolve((f1, f2), (x1, x2), (-1, 1)))
```

    Matrix([[-1.19287309935246], [1.27844411169911]])

7. Ordinary Differential Equations (ODE)

ODE can be solved in the following way (More information at http://docs.sympy.org/latest/modules/solvers/ode.html):

```python
f = Function("f")
diff_equation = Eq(f(x).diff()/f(x), 3/x) # In other words: y'/y = 3/x
diff_result = dsolve(diff_equation, f(x))
diff_result
```

It is also possible to classify ODEs wth SymPy:

```python
classify_ode(diff_equation, f(x))
```

    ('separable',
     '1st_exact',
     '1st_linear',
     '1st_homogeneous_coeff_best',
     '1st_homogeneous_coeff_subs_indep_div_dep',
     '1st_homogeneous_coeff_subs_dep_div_indep',
     'lie_group',
     'separable_Integral',
     '1st_exact_Integral',
     '1st_linear_Integral',
     '1st_homogeneous_coeff_subs_indep_div_dep_Integral',
     '1st_homogeneous_coeff_subs_dep_div_indep_Integral')

For Partial Differental Equations, look here: http://docs.sympy.org/latest/modules/solvers/pde.html

8. Conversion of SymPy representations into Python code

You can also convert SymPy functions to Python functions to use them directly in your programs:

```python
result_function = diff_result.rhs
result_function
```

```python
python_function = lambdify(('C1', x), result_function)
python_function(1, 2) # C1=1, x=2.
```

If you use Cython (http://cython.org/), you can use more efficient solutions to pythonize your SymPy functions, see http://docs.sympy.org/latest/modules/numeric-computation.html for more information.

PSB 2017


```python
import sympy as sm
import scipy as sp
```

```python
sm.init_printing()
```

### Calculate determinat

```python
m11, m12, m13, m14, m21, m22, m23, m24, m31, m32, m33, m34, m41, m42, m43, m44 \
= sm.symbols("m11, m12, m13, m14, m21, m22, m23, m24, m31, m32, m33, m34, m41, m42, m43, m44")
```

```python
A = sm.Matrix([[m11, m12, m13, m14],
               [m21, m22, m23, m24],
               [m31, m32, m33, m34],
               [m41, m42, m43, m44]])
```

```python
A.det()
```

```python
A_num = A.subs({m11:3, m12:7, m13:-2, m14:4,
                m21:-3, m22:-2, m23:6, m24:-4,
                m31:5, m32:5, m33:-3, m34:2,
                m41:2, m42:6, m43:-5, m44:3})
A_num
```

$\displaystyle \left[\begin{matrix}3 & 7 & -2 & 4\\-3 & -2 & 6 & -4\\5 & 5 & -3 & 2\\2 & 6 & -5 & 3\end{matrix}\right]$

```python
A_num.det()
```

### Inverse matrix

```python
A_num.inv()
```

$\displaystyle \left[\begin{matrix}- \frac{8}{303} & - \frac{5}{101} & \frac{82}{303} & - \frac{64}{303}\\\frac{7}{303} & \frac{17}{101} & \frac{4}{303} & \frac{56}{303}\\\frac{67}{303} & \frac{4}{101} & - \frac{5}{303} & - \frac{70}{303}\\\frac{103}{303} & - \frac{24}{101} & - \frac{71}{303} & - \frac{85}{303}\end{matrix}\right]$

### Systems of equations

```python
x1, x2, x3, x4 = sm.symbols("x1, x2, x3, x4")
```

```python
sm.solve([x1 + 5*x2 + 4*x3 + 3*x4 - 1,
         2*x1 - x2 + 2*x3 - x4,
         5*x1 + 3*x2 + 8*x3 + x4 - 1],
         [x1, x2, x3, x4])
```

```python
sm.solve([x1 + x2 - 2*x3 + x4 - 1,
          x1 - 3*x2 + x3 + x4,
          4*x1 - x2 - x3 - x4 - 1,
          4*x1 + 3*x2 - 4*x3 - x4 - 2], [x1, x2, x3, x4])
```

### Limits

```python
x, y, b, c = sm.symbols("x, y, b, c")
```

```python
f_1 = (1 - sm.cos(5*x))/(x**2)
```

```python
sm.limit(f_1, x, 0)
```

```python
f_2 = ((x+1)/(x+b))**(x+c)
```

```python
sm.limit(f_1, x, sm.oo)
```

### Derivatives

```python
a, b = sm.symbols("a, b")
```

```python
sm.diff ((sm.asin(x)/x)**(x**3), x)
```

```python
sm.diff ((sm.ln(sm.sqrt((a+sm.sin(x))/sm.tan(x)-b))), x)
```

### Mixed partial derivatives

```python
n = sm.Symbol("n")
```

```python
f1 = x**(2*n)*sm.sin(y)
```

```python
f2 = (sm.E**y)*sm.sin(n*x**2)
```

```python
sm.diff (f1, x, 2, y, 3)
```

```python
sm.diff (f2, x, 2, y, 3)
```

### Integrals

```python
r = sm.Symbol ("r")
```

```python
f = (sm.sin(sm.cbrt(x)))/sm.cbrt(x**2)
g = sm.sqrt(r**2 - x**2, x)
g
```

```python
sm.integrate (f, x)
```

```python
sm.integrate (g, (x, 0, r))
```

$\displaystyle \begin{cases} - \int\limits_{r}^{0} \begin{cases} - \frac{i r}{\sqrt{-1 + \frac{x^{2}}{r^{2}}}} + \frac{3 i x^{2}}{2 r \sqrt{-1 + \frac{x^{2}}{r^{2}}}} + \frac{i x^{2}}{2 r \left(-1 + \frac{x^{2}}{r^{2}}\right)^{\frac{3}{2}}} - \frac{i x^{4}}{2 r^{3} \left(-1 + \frac{x^{2}}{r^{2}}\right)^{\frac{3}{2}}} & \text{for}\: \frac{\left|{x^{2}}\right|}{r^{2}} > 1 \\\frac{r \sqrt{1 - \frac{x^{2}}{r^{2}}}}{2} + \frac{r}{2 \sqrt{1 - \frac{x^{2}}{r^{2}}}} - \frac{x^{2}}{2 r \sqrt{1 - \frac{x^{2}}{r^{2}}}} & \text{otherwise} \end{cases}\, dx & \text{for}\: r < 0 \\\int\limits_{0}^{r} \begin{cases} - \frac{i r}{\sqrt{-1 + \frac{x^{2}}{r^{2}}}} + \frac{3 i x^{2}}{2 r \sqrt{-1 + \frac{x^{2}}{r^{2}}}} + \frac{i x^{2}}{2 r \left(-1 + \frac{x^{2}}{r^{2}}\right)^{\frac{3}{2}}} - \frac{i x^{4}}{2 r^{3} \left(-1 + \frac{x^{2}}{r^{2}}\right)^{\frac{3}{2}}} & \text{for}\: \frac{\left|{x^{2}}\right|}{r^{2}} > 1 \\\frac{r \sqrt{1 - \frac{x^{2}}{r^{2}}}}{2} + \frac{r}{2 \sqrt{1 - \frac{x^{2}}{r^{2}}}} - \frac{x^{2}}{2 r \sqrt{1 - \frac{x^{2}}{r^{2}}}} & \text{otherwise} \end{cases}\, dx & \text{otherwise} \end{cases}$

### Sum and product

```python
k = sm.Symbol("k")
```

```python
s = 1/((2*n - 1)*(2*n + 1))
```

```python
sm.Sum(s, (n, 1, sm.oo)).evalf()
```

```python
p = sm.sin(k)
```

```python
sm.Product(p, (k, 1, 100)).evalf()
```

### Graphs

```python
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
```

```python
matplotlib.rcParams.update({'axes.facecolor': 'white',
                            'axes.labelcolor': 'darkslategray',
                            'axes.titlecolor': 'darkslategray',
                            'text.color': 'darkslategray',
                            'font.size': '15',
                            'figure.titleweight':'heavy',
                            'grid.linestyle': 'dashed',
                            'legend.fontsize': '15',
                            'legend.loc': 'lower left'})
```

```python
x = np.linspace(-2, 3, 1000)
f = -np.exp(-(x - 0.7)**2)
```

```python
fig, ax = plt.subplots()

ax.plot(x, f, label = r'${e^{-(x - 0.7)}}^2$')
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
ax.legend()
ax.grid(True)
plt.title('First graph')
plt.show()
```

```python
x = np.linspace(-10, 10, 1000)
y = np.linspace(-10, 10, 1000)

z = np.sin(np.sqrt(x**2 + y**2))/np.sqrt(x**2 + y**2 + 0.1)
```

```python
fig, ax = plt.subplots()

ax.plot(x, z)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
ax.grid(True)
plt.title('Second graph')
plt.show()
```

```python
x = sm.Symbol("x")
f = -sm.exp(-(x - 0.7)**2)
f_diff = sm.diff(f, x)
x_min = sm.solve(f_diff, x)
```

```python
x = np.linspace(-10, 10, 1000)
y = -(1.4-2*x)*np.e**(-(x-0.7)**2)
```

```python
fig, ax = plt.subplots()

ax.plot(x, y)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
ax.grid(True)
plt.title('derivative')
plt.show()
```

As you can see on the graph, x_min is minimum of function

```python

```


CompEcon Toolbox:

DemApp05

Chebychev polynomial and spline approximantion of various functions

Randall Romero Aguilar, PhD

This demo is based on the original Matlab demo accompanying the  Computational Economics and Finance 2001 textbook by Mario Miranda and Paul Fackler.

Last updated: 2020-Sep-08

## About

Demonstrates Chebychev polynomial, cubic spline, and linear spline approximation for the following functions
\begin{align}
    y &= 1 + x + 2x^2 - 3x^3 \\
    y &= \exp(-x) \\
    y &= \frac{1}{1+25x^2} \\
    y &= \sqrt{|x|}
\end{align}

## Initial tasks

```python
if 'google.colab' in str(get_ipython()):
    print("This notebook is running on Google Colab. Installing the compecon package.")
    !pip install compecon
```

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from compecon import BasisChebyshev, BasisSpline, nodeunif
```

### Functions to be approximated

```python
funcs = [lambda x: 1 + x + 2 * x ** 2 - 3 * x ** 3,
         lambda x: np.exp(-x),
         lambda x: 1 / ( 1 + 25 * x ** 2),
         lambda x: np.sqrt(np.abs(x))]

fst = ['$y = 1 + x + 2x^2 - 3x^3$', '$y = \exp(-x)$',
       '$y = 1/(1+25x^2)$', '$y = \sqrt{|x|}$']
```

Set degree of approximation and endpoints of approximation interval

```python
n = 7   # degree of approximation
a = -1  # left endpoint
b = 1   # right endpoint
```

Construct uniform grid for error ploting

```python
x = np.linspace(a, b, 2001)
```

```python
def subfig(f,  title):

    # Construct interpolants
    C = BasisChebyshev(n, a, b, f=f)
    S = BasisSpline(n, a, b, f=f)
    L = BasisSpline(n, a, b, k=1, f=f)

    data = pd.DataFrame({
        'actual': f(x),
        'Chebyshev': C(x),
        'Cubic Spline': S(x),
        'Linear Spline': L(x)},
        index = x)

    fig1, axs = plt.subplots(2,2, figsize=[12,6], sharex=True, sharey=True)
    fig1.suptitle(title)
    data.plot(ax=axs, subplots=True)

    errors = data[['Chebyshev', 'Cubic Spline']].subtract(data['actual'], axis=0)

    fig2, ax = plt.subplots(figsize=[12,3])
    fig2.suptitle("Approximation Error")
    errors.plot(ax=ax)

```

## Polynomial
$y = 1 + x + 2x^2 - 3x^3$

```python
subfig(lambda x: 1 + x + 2*x**2 - 3*x**3, '$y = 1 + x + 2x^2 - 3x^3$')
```

## Exponential
$y = \exp(-x)$

```python
subfig(lambda x: np.exp(-x),'$y = \exp(-x)$')
```

## Rational
$y = 1/(1+25x^2)$

```python
subfig(lambda x: 1 / ( 1 + 25 * x ** 2),'$y = 1/(1+25x^2)$')
```

## Kinky
$y = \sqrt{|x|}$

```python
subfig(lambda x: np.sqrt(np.abs(x)), '$y = \sqrt{|x|}$')
```

notebooks/app/05 Chebychev polynomial and spline approximantion of various functions.ipynb