# POLINOMIO DE INTERPOLACIÓN DE HERMITE
Grupo $\iota$ (iota)<br/>
Importante: Para correr los códigos, es necesario tener instalado lo siguiente:
- Python 2.7 o superior
- Numpy (para el manejo de arreglos optimizados)
- Sympy (para los polinomios)
- Matplotlib (para las gráficas)

Consideremos el problema de buscar un polinomio que cumpla con las siguientes características:
- Sean $x_0, . . . , x_n$ números distintos pertenecientes al intervalo $[a,b]$ de $f$ en $x_i$, un polinomio $p(x_i)$ que aproxime a las mismas imágenes de $f(x_i)$ con $i=0,1,2,...,n$

$$p(x_i)=f(x_i)$$

- Además, $p(x_i)$ debe aproximar a las imágenes de la primera derivada de $f(x_i)$

$$p'(x_i)=f'(x_i)$$

El polonomio que cumple con esas características es el **_Polinomio de Interpolación de Hermite_**.

## TEOREMA:

Si $f ∈ C^1 [a, b] $ y $x_0, . . . , x_n ∈ [a, b]$ son distintos, el **único** polinomio de menor grado que concuerda con $f$ y $f'$ en $x_0,x_1,...,x_n$ es el polinomio de Hermite de grado a lo sumo $2n + 1$ dado por

$$H_{2n+1}(x) = \sum_{j=0}^{n} f(x_j)H_{n,j}(x) + \sum_{j=0}^{n} f'(x_j)\hat{H}_{n,j}(x)$$

Siendo:

$$H_{n,j}(x) = [1 - 2(x-x_j)L'_{n,j}(x_j)][L_{n,j}(x)]^2$$
$$\hat{H}_{n,j}(x) = (x-x_j)[L_{n,j}(x)]^2$$

Donde cada $L_{n,j}(x)$ son el $i$-ésimo coeficiente del polonomio de Lagrange de grado n.

## DEMOSTRACIÓN:

- ### $H_{2n + 1}(x) = f(x)$

Al igual que el _Lagrange_, Hermite es un polinomio $H_{2n + 1}(x)$ que interpola los mismos puntos que una función $f(x)$, por eso se le denomina **_polinomio osculante_**, ya que no tendrá específicamente la misma forma que $f(x)$, si no que solo "besará" los puntos dados de la función.

Por lo tanto, recordemos que en _Lagrange_:

$$L_{n,j}(x_i) = \begin{Bmatrix}0, i \neq j \\1, i=j\end{Bmatrix}$$

Entonces, cuando $i \neq j$:

$$H_{n,j}(x_i) = 0$$
$$y$$
$$\hat{H}_{n,j}(x_i)=0$$

Por lo que para cada $i$:

$$H_{n,i}(x_i) = [1 - 2(x_i-x_i)L'_{n,i}(x_i)]\cdot 1^2 = 1$$
$$y$$
$$\hat{H}_{n,i}(x_i) = (x_i-x_i)\cdot1^2 = 0$$

Sustituyendo:

$$H_{2n+1}(x_i) = \sum_{\substack{j=0 \\j\neq i}} ^{n} f(x_j) \cdot 0 + f(x_i) \cdot 1 + \sum_{j=0}^{n} f'(x_j) \cdot 0$$
$$H_{2n+1}(x_i) = f(x_i)$$
$$lqqd$$

- ### $H'_{2n + 1}(x) = f'(x)$

Realizando las derivadas obtenemos:

$$H'_{n,j}(x_i) = -2L'_{n,j}(x_i)\cdot[L_{n,j}(x_i)]^2 + 2\cdot[1 - 2(x_i-x_j)L'_{n,j}(x_i)]\cdot[L_{n,j}(x_i)]\cdot[L'_{n,j}(x_i)]$$
$$H'_{n,j}(x_i) = -2L'_{n,j}(x_i) + 2L'_{n,j}(x_i)$$
$$H'_{n,j}(x_i) = 0$$

Por lo tanto, $H'_{n,j}(x_i) = 0$ para todas las $i$ y $j$, finalmente:

$$\hat{H}'_{n,j}(x_i) = [L_{n,j}(x_i)]^2 + (x_i - x_j)2L_{n,j}(x_i)L'_{n,j}(x_i)$$
$$\hat{H}'_{n,j}(x_i) = 0$$

Sustituyendo:

$$H'_{2n+1}(x_i) = \sum_{j=0}^{n} f(x_j) \cdot 0 + \sum_{\substack{j=0\\j\neq i}}^{n} f'(x_j) \cdot 0 + f'(x_i) \cdot 1$$
$$H'_{2n+1}(x_i) = f'(x_i)$$
$$lqqd$$

## VENTAJAS:
- Además de encontrar con Hermite un polinomio que tenga las mismas imágenes que la función original, sus derivadas también son las mismas imágenes que las derivadas de la función original.
- Es fácil de calcular ya que garantiza la posición de los frames al ser una interpolación además de brindar la certeza de que la tangente de la curva generada es continua a lo largo de multiples segmentos.
- La precisión será más exacta, ya que las rectas tangentes serán las mismas que las de la función original.

## DESVENTAJAS:
- Ya que el grado del polinomio que cumple con los requisitos es de grado $2n + 1$, el grado n para obtener el polinomio de Hermite siempre será más alto de lo necesario.
- Requiere la disponibilidad de las primeras derivadas, lo cual en muchos casos son desconocidas.
- Similar a Lagrange, Hermite requiere que $f'(x_i)$ esté dentro de $0 < i < 1$, y no todas las funciones son capaces de cumplirlo.

## EJERCICIOS:

- ### Ejercicio 1

Use el polinomio de Hermite que concuerda con los datos listados en la tabla para encontrar una aproximación de $f(1.5)$.

|$k$|$x_k$|$f(x_k)$   |$f'(x_k)$   |
|---|-----|-----------|------------|
| 0 | 1.3 | 0.6200860 | −0.5220232 |
| 1 | 1.6 | 0.4554022 | −0.5698959 |
| 2 | 1.9 | 0.2818186 | −0.5811571 |

**Solución:**
Primero calculamos los polinomios de Lagrange y sus derivadas.

$$L_{2,0}(x)= \frac{(x-x_1)(x-x_2)}{(x_0-x_1)(x_0-x_2)}= \frac{50}{9}x^2-\frac{175}{9}x+\frac{152}{9},   \ \ \ \ \ \ \ \ \ L'_{2,0}(x)=\frac{100}{9}x-\frac{175}{9};$$

$$L_{2,1}(x)= \frac{(x-x_0)(x-x_2)}{(x_1-x_0)(x_1-x_2)}= \frac{-100}{9}x^2-\frac{320}{9}x+\frac{247}{9},  \ \ \ \ \ \ \ \ \ L'_{2,1}(x)=\frac{-200}{9}x-\frac{320}{9};$$

$$L_{2,2}(x)= \frac{(x-x_0)(x-x_1)}{(x_2-x_0)(x_2-x_1)}= \frac{50}{9}x^2-\frac{145}{9}x+\frac{104}{9},  \ \ \ \ \ \ \ \ \ L'_{2,2}(x)=\frac{100}{9}x-\frac{145}{9}.$$

Los polinomios $H_{2,j}(x)$ y $\hat{H}_{2,j}(x)$ son entonces

$$H_{2,0}(x)=[1-2(x-1.3)(-5)](\frac{50}{9}x^2-\frac{175}{9}x+\frac{152}{9})^2 = (10x-12)(\frac{50}{9}x^2-\frac{175}{9}x+\frac{152}{9})^2,$$

$$H_{2,1}(x)=1.(\frac{-100}{9}x^2+\frac{320}{9}x-\frac{247}{9})^2,$$

$$H_{2,2}(x)=10(2-x)(\frac{50}{9}x^2-\frac{145}{9}x+\frac{104}{9})^2,$$

$$\hat{H}_{2,0}(x)=(x-1.3)(\frac{50}{9}x^2-\frac{175}{9}x+\frac{152}{9})^2,$$

$$\hat{H}_{2,1}(x)=(x-1.6)(\frac{-100}{9}x^2+\frac{320}{9}x-\frac{247}{9})^2,$$

$$\hat{H}_{2,2}(x)=(x-1.9)(\frac{50}{9}x^2-\frac{145}{9}x+\frac{104}{9})^2.$$

Finalmente, multiplicamos los $H_{2,j}(x)$ por las imagenes en "y" $f(x_k)$ y los $\hat{H}_{2,j}(x)$ por los valores de las derivadas  $f'(x_k)$

$$H_5(x) = 0.6200860 H_{2,0}(x) + 0.4554022 H_{2,1}(x) + 0.2818186 H_{2,2}(x) − 0.5220232 \hat{H}_{2,0}(x) - 0.5698959 \hat{H}_{2,1}(x) - 0.5811571 \hat{H}_{2,2}(x)$$

$$H_5(1.5) = 0.6200860 (\frac{4}{27}) + 0.4554022 (\frac{64}{81}) + 0.2818186 (\frac{5}{81}) − 0.5220232 (\frac{4}{405}) - 0.5698959 (\frac{-32}{405})  - 0.5811571 (\frac{-2}{405})$$

$$H_5(1.5) = {\color{green}{0.5118277}}$$

- ### Ejercicio 1.1: Realizando el ejercicio anterior con python

```python
from sympy import *
import matplotlib.pyplot as plt
import numpy as np

x = Symbol("x")

'''
n : valor del grado del polinomio 2n + 1
xs : valores de x
ys : valores de y
dys : valores de la primera derivada
'''

def hermite(n, xs, ys, dys):
    h = []
    hatH = []

    for i in range(0, (n+1)//2):
        l = lagrangeFactor(n, xs, i)
        d = diff(l)
        f = (1-2*(x-xs[i])*d.subs(x, xs[i]))*l**2
        hatF = (x-xs[i])*l**2
        h.append(f)
        hatH.append(hatF)

    p = 0

    for i in range(0, (n+1)//2):
        p += ys[i]*h[i]

    for i in range(0, (n+1)//2):
        p += dys[i]*hatH[i]

    return p

'''
n : valor del grado del polinomio 2n + 1
xs : valores de x
k : valor iterativo
'''

def lagrangeFactor(n, xs, k):
    L = 1
    for i in range(0, (n+1)//2):
        if (i != k):
            L *= (x - xs[i])/(xs[k] - xs[i])
    return L

xs = [1.3, 1.6, 1.9]
ys = [0.6200860, 0.4554022, 0.2818186]
dys = [-0.5220232, -0.5698959, -0.5811571]

pol = hermite(5, xs, ys, dys)
pol.expand()

```

$\displaystyle - 0.00277469135824404 x^{5} + 0.024031790125548 x^{4} - 0.0145560802536693 x^{3} - 0.235216169740852 x^{2} - 0.00822922346583255 x + 1.00194406469399$

Por lo tanto, al sustituír $H_{2n + 1}(1.5)$ obtenemos:

```python
pol.subs(x, 1.5)

```

$\displaystyle 0.511827701728394$

Graficando los puntos y el polinomio obtenemos:

```python
# Datos
a = np.min(xs[0]-1)
b = np.max(xs[len(xs)-1]+1)
hx = np.linspace(a,b,200)
py = lambdify(x,pol)
hy = py(hx)

# Gráfica
plt.plot(xs, ys, 'o', label='Puntos')
plt.plot(hx,hy,label='Polinomio')
plt.legend()
plt.xlabel('xi')
plt.ylabel('fi')
plt.title('Lagrange')
plt.show()
```

## INTERPOLACIÓN DE HERMITE CON DIFERENCIAS DIVIDIDAS

Las diferencias divididas generalizadas se construyen de igual manera que las _Diferencias Divididas de Newton_, con la única diferencia que ahora se necesita replicar nuestras muestras $x_i$ tantas veces como las derivadas que poseamos, en este caso se estudiara únicamente cuando se conoce la primera derivada.

De igual manera que en la interpolación de Lagrange la interpolación de Hermite con el grado $2n + 1$ se escribirá inmediatamente calculadas las _Diferencias Divididas_.

$$H_{2n+1}(x) = f[x_0] + f[x_0,x_0](x-x_0)+...+f[x_0,x_0,...,x_n,x_n](x-x_0)^2(x-x_{n-1})^2(x-x_n)$$

Aparentemente se nota que $f[x_i,x_i]$ no estan definida puesto que

$$f[x_i, x_i] = \frac{f[x_i]-f[x_i]}{x_i-x_i} = \frac{0}{0}$$

Sin embargo, podemos tomar un paso infinitisimal $h$ para el termino resultando en

$$f[x_i+h, x_i] = \frac{f[x_i+h]-f[x_i]}{x_i+h-x_i} = \frac{f[x_i+h]-f[x_i]}{h}$$

Si se toma el limite cuando $h \rightarrow 0$ se tendrá

$$f[x_i, x_i] = f'(x_i)$$

Con lo anterior, es posible concluir que la nueva tabla de diferencias divididas es la siguiente:

| $z$      | $f(z)$            | $1^{ra} Diferencia Dividida$                 |$2^{da} Diferencia Dividida$                          |
|----------|-------------------|----------------------------------------------|------------------------------------------------------|
|$z_0=x_0$ |$ f[z_0) = f(x_0)$ |                                              |                                                      |
|          |                   |$f[z_0,z_1] = f'(x_0)$                        |                                                      |
|$z_1=x_0$ |$ f[z_1] = f(x_0)$ |                                              |$f[z_0,z_1,z_2]=\frac{f[z_1,z_2]-f[z_0,z_1]}{z_2-z_0}$|
|          |                   |$f[z_1,z_2] = \frac{f[z_2]-f[z_1]}{z_2-z_1}$  |                                                      |
|$z_2=x_1$ |$ f[z_2] = f(x_1)$ |                                              |$f[z_1,z_2,z_3]=\frac{f[z_2,z_3]-f[z_1,z_2]}{z_3-z_1}$|
|          |                   |$f[z_2,z_3] = f'(x_1)$                        |                                                      |
|$z_3=x_1$ |$ f[z_3] = f(x_1)$ |                                              |$f[z_2,z_3,z_4]=\frac{f[z_3,z_4]-f[z_2,z_3]}{z_4-z_2}$|
|          |                   |$f[z_3,z_4] = \frac{f[z_4]-f[z_3]}{z_4 - z_5}$|                                                      |
|$z_4=x_2$ |$ f[z_4] = f(x_2)$ |                                              |$f[z_3,z_4,z_5]=\frac{f[z_4,z_5]-f[z_3,z_4]}{z_5-z_3}$|
|          |                   |$f[z_4,z_5] = f'(x_2)$                        |                                                      |
|$z_5=x_2$ |$ f[z_5] = f(x_2)$ |                                              |                                                      |

- ### Ejercicio 2: Realizar el ejercicio 1 utilizando Interpolación de Hermite con Diferencias Divididas

Usando los datos que se proporcionan en la tabla y el método de diferencias divididas de Newton, determine la aproximación polinomial de Hermite en $x=1.5$.

|$k$|$x_k$|$f(x_k)$   |$f'(x_k)$   |
|---|-----|-----------|------------|
| 0 | 1.3 | 0.6200860 | −0.5220232 |
| 1 | 1.6 | 0.4554022 | −0.5698959 |
| 2 | 1.9 | 0.2818186 | −0.5811571 |

**Solución:** Primero calcularemos las diferencias divididas:

| $z$                 | $f(z)$                    |$1^{ra}$                    |$2^{da}$     |$3^{ra}$    |$4^{ta}$    |$5^{ta}$     |
|---------------------|---------------------------|----------------------------|-------------|------------|------------|-------------|
|${\color{orange}{1.3}}$|${\color{orange}{0.6200860}}$|                            |             |            |            |             |
|                     |                           |${\color{orange}{-0.5220232}}$|             |            |            |             |
|${\color{orange}{1.3}}$|${\color{orange}{0.6200860}}$|                            |$-0.0897427$ |            |            |             |
|                     |                           |$-0.5489460$                |             |$0.0663657$ |            |             |
|${\color{orange}{1.6}}$|${\color{orange}{0.4554022}}$|                            |$-0.0698330$ |            |$0.0026663$ |             |
|                     |                           |${\color{orange}{−0.5698959}}$|             |$0.0679655$ |            |$-0.0027738$ |
|${\color{orange}{1.6}}$|${\color{orange}{0.4554022}}$|                            |$-0.0290537$ |            |$0.0010020$ |             |
|                     |                           |$-0.5786120$                |             |$0.0685667$ |            |             |
|${\color{orange}{1.9}}$|${\color{orange}{0.2818186}}$|                            |$-0.0084837$ |            |            |             |
|                     |                           |${\color{orange}{−0.5811571}}$|             |            |            |             |
|${\color{orange}{1.9}}$|${\color{orange}{0.2818186}}$|                            |             |            |            |             |

Las entradas en color naranja en las primeras tres columnas de la tabla son los datos que se proporcionaron por el ejercicio. Las entradas restantes en esta tabla se generan con la fórmula de diferencias divididas estándar. Por ejemplo, para la segunda entrada en la tercera columna usamos la segunda entrada 1.3 en la segunda columna y la primera entrada 1.6 en esa columna para obtener:

$$ \frac{0.4554022 − 0.6200860}{1.6-1.3} = −0.5489460 $$

Para la primera entrada en la cuarta columna, usamos la primera entrada 1.3 en la tercera columna y la primera entrada 1.6 en esa columna para obtener:

$$ \frac{−0.5489460 − (−0.5220232)}{1.6 − 1.3} = −0.0897427 $$

El valor del polinomio de Hermite en 1.5 es:

$$H_5(1.5) = f[1.3]+f'(1.3)(1.5 − 1.3) + f[1.3, 1.3, 1.6](1.5 − 1.3)^2+f[1.3, 1.3, 1.6, 1.6](1.5 − 1.3)^2(1.5 − 1.6)$$
$$\ \ \ \ \ \ \ \ \ +f[1.3, 1.3, 1.6, 1.6, 1.9](1.5 − 1.3)^2(1.5 − 1.6)^2+f[1.3, 1.3, 1.6, 1.6, 1.9, 1.9](1.5 − 1.3)^2(1.5 − 1.6)^2(1.5 − 1.9)$$

$$H_5(1.5) =0.6200860 + (−0.5220232)(0.2) + (−0.0897427)(0.2)^2+ 0.0663657(0.2)^2(−0.1) + 0.0026663(0.2)^2(−0.1)^2+ (−0.0027738)(0.2)^2(−0.1)^2(−0.4)$$
$$H_5(1.5) = {\color{green}{0.5118277}}$$

- ### Ejercicio 2.1: Realizar el ejercicio 2 utilizando Interpolación de Hermite con Diferencias Dividas

```python
def graphic(pol, xi, fi, x):
    # polinomio para evaluacion numérica
    px = lambdify(x,pol)

    # Puntos para la gráfica
    samples = 200
    a = np.min(xi-1)
    b = np.max(xi+1)
    pxi = np.linspace(a,b,samples)
    pfi = px(pxi)

    np.set_printoptions(precision = 4)

    # Gráfica
    plt.plot(xi,fi,'o', label = 'Puntos')
    plt.plot(pxi,pfi, label = 'Polinomio')
    plt.legend()
    plt.xlabel('xi')
    plt.ylabel('fi')
    plt.title('Diferencias Divididas Simple Generalizada')
    plt.show()

def getSimplePolynomial(factors, fi, xi, n, x ):
    pol = fi[0]
    for j in range(1,n,1):
        factor = factors[j-1]
        term = 1
        for k in range(0,j,1):
            term = term*(x-xi[k])
        pol = pol + term*factor

    return pol.expand()

def fillTable(table, n, m, xi, dfi):
    d = n-1
    j = 3

    tmp_table = table
    # Calcula tabla, inicia en columna 3
    while (j < m):

        # cada fila de columna
        i = 0
        step = j-2 # inicia en 1
        derivate_iterator = 0
        while (i < d):
            denominador = (xi[i+step]-xi[i])
            numerador = tmp_table[i+1,j-1]-tmp_table[i,j-1]

            if denominador == 0 and numerador == 0:
                tmp_table[i, j] = dfi[derivate_iterator]
                derivate_iterator = derivate_iterator + 1
            else :
                tmp_table[i,j] = numerador/denominador
            i = i+1
        d = d - 1
        j = j+1

    return tmp_table

def duplicateArray(arr):
    duplicates_values = np.array([])
    for number in arr:
        duplicates_values = np.append(duplicates_values, np.repeat(number, 2))

    return duplicates_values

def newtonHermite(xi, fi, dfi) :
    duplicate_xi = duplicateArray(xi)
    duplicate_fi = duplicateArray(fi)
    array_length = len(duplicate_xi)

    ki = np.arange(0,array_length,1)
    table = np.concatenate(([ki],[duplicate_xi],[duplicate_fi]),axis=0)
    table = np.transpose(table)

    dfinita = np.zeros(shape=(array_length,array_length),dtype=float)
    table = np.concatenate((table,dfinita), axis=1)
    [array_length,m] = np.shape(table)

    table = fillTable(table, array_length, m, duplicate_xi, dfi)

    newton_factors = table[0,3:]
    array_length = len(dfinita)
    x = Symbol('x')

    polynomial = getSimplePolynomial(newton_factors,  duplicate_fi, duplicate_xi, array_length, x)
    graphic(polynomial, duplicate_xi, duplicate_fi, x)

    polynomial

    return polynomial

hPol = newtonHermite(np.array([1.3,1.6, 1.9]), np.array([  0.6200860, 0.4554022, 0.2818186]),np.array([ -0.5220232, -0.5698959, -0.5811571]))
```

```python
hPol
```

$\displaystyle - 0.00277469135798944 x^{5} + 0.0240317901231809 x^{4} - 0.0145560802460553 x^{3} - 0.235216169754413 x^{2} - 0.00822922345577151 x + 1.00194406469105$

Por lo tanto, al sustituír $H_{2n + 1}(1.5)$ obtenemos:

```python
hPol.subs(x,1.5)
```

$\displaystyle 0.511827701728395$

- ### Ejercicio 3:

Use la interpolación de Hermite en en python que concuerda con los datos listados en la tabla para encontrar una aproximación de $f(1.2)$.

|$k$|$x_k$| $f(x_k)$  | $f'(x_k)$  |
|---|-----|-----------|------------|
| 0 | -1  |    -2     |     18     |
| 1 |  0  |     2     |     -1     |
| 2 |  1  |     6     |     18     |

- #### Usando la función propuesta en el ejercicio 1.1

```python
xs = [-1, 0, 1]
ys = [-2,2,6]
dys = [18,-1,18]

pol = hermite(5, xs, ys, dys)
pol.expand()

```

$\displaystyle 2 x^{5} + 3 x^{3} - x + 2$

Por lo tanto, al sustituír $H_{2n + 1}(0.5)$ obtenemos:

```python
pol.subs(x,0.5)
```

$\displaystyle 1.9375$

Graficando los puntos y el polinomio obtenemos:

```python
# Datos
a = np.min(xs[0]-1)
b = np.max(xs[len(xs)-1]+1)
hx = np.linspace(a,b,101)
py = lambdify(x,pol)
hy = py(hx)

# Gráfica
plt.plot(xs, ys, 'o', label='Puntos')
plt.plot(hx,hy,label='Polinomio')
plt.legend()
plt.xlabel('xi')
plt.ylabel('fi')
plt.title('Lagrange')
plt.show()
```

- #### Usando la función del ejercicio 2.1

```python
hPol = newtonHermite(np.array([-1,0, 1]), np.array([  -2, 2, 6]),np.array([ 18, -1, 18]))
```

```python
hPol
```

$\displaystyle 2.0 x^{5} + 3.0 x^{3} - 1.0 x + 2.0$

Por lo tanto, al sustituír $H_{2n + 1}(0.5)$ obtenemos:

```python
hPol.subs(x,0.5)
```

$\displaystyle 1.9375$

## COTA DEL ERROR:

Sea $f$ una función continua sobre $[a,b]$, tal que las derivadas de $f$ de orden $\leqslant 2n + 2$ existen y son continuas en $[a,b]$. Si $p_{2n+1} \in P_{2n+1}$ es el polinomio que interpola a la función $f$ en los $n +1$ valores distintos $x_0,x_1,...,x_n \in [a,b]$, entonces para todo $x \in [a,b]$, existe $\xi = \xi(x) \in \ ]a,b[$, tal que:

$$f(x) = P_{2n+1}(x) + \frac{f^{(2n+2)}(\xi)}{(2n + 2)!}[\pi_{n+1}(x)]^2$$

Donde:

$$\pi_{n+1}(x) = \prod_{j=0}^{n} (x-x_j) = (x-x_0)(x-x_1)...(x-x_n)$$

Ademas, para $M_{2n+2} = \max_{z \in [a,b]}|f^{(2n+2)}(z)|$, se cumple que:

$$|f(x)- P_{2n+1}(x)| \leqslant \frac{M_{2n+2}}{(2n+2)!}[\pi_{n+1}(x)]^2  \ \ \ \ \ \ \ \ \  \forall x \in [a,b].$$

- ### Ejemplo

Considerando $f(x) = ln(x)$ y $p_3 \in P_3$ el polinomio de interporlación de Hermite para los nodos $x_0 = 1$ y $x_1 = 2$.
Determine una cota de error del polinomio $p_3$.

La cota del error del polinomio de interporlación de Hermite está dado por:

Al tener dos nodos, $n=1$, por lo tanto:

$$|f(x) - p_3 (x)| \leqslant \frac{M_4}{4!}[\pi_4 (x)]^2 = \frac{M_4}{24}(x-1)^2(x-2)^2$$

Observe que:

$$f'(x) = \frac{1}{x}$$
$$f''(x) = \frac{-1}{x^2}$$
$$f'''(x) = \frac{2}{x^3}$$
$$f^{(4)}(x) = \frac{-6}{x^4}$$

Buscando el maximo:

$$M_4 = \max_{x \in [1,2]}|f^{(4)}(x) | = \max_{x \in [1,2]}\{\frac{6}{x^4}\} = \frac{6}{1^4} = 6 $$

Mientras que:

$$[\pi_4(x)]^2 = (x^2-2x+1)(x^2-4x+4)$$
$$[\pi_4(x)]^2 = x^4-6x^3+13x^2-12x+4$$

Por otro lado:

$$0 = [[\pi_4(x)]^2]'$$
$$0 = 4x^3 - 18x^2 + 26x - 12$$
$$0 = 2(x-1)(2x-3)(x-2)$$

Lo que nos permite conocer los puntos criticos que son: $x =1$, $x = \frac{3}{2}$, $ x = 2$

Evaluamos en los extremos y en los puntos críticos:
- _Observar que el valor máximo siempre será positivo porque está elevado al cuadrado_

$$[\pi_4(1)]^2 = 0$$
$$[\pi_4(1.5)]^2 = \frac{1}{16}$$
$$[\pi_4(2)]^2 = 0$$

Se deduce que:

$$[\pi_4(x)]^2 \leqslant \frac{1}{16} \ \ \ \ \ \ \ \ \  \forall x \in [1,2].$$

Por lo tanto se obtiene el valor de la cota:

$$|f(x) - p_3(x)| \leqslant \frac{6}{24} . \frac{1}{16} = {\color{green}{\frac{1}{64}}} \ \ \ \ \ \ \ \ \  \forall x \in [1,2].$$

## REFERENCIAS:

_Burden, Richard & Faires, J. Douglas (2010). Hermite Interpolation en Numerical Analysis, novena edición (pp. 136-144). Cengage Learning._ <br/>
_Trejos, C. V. (2020, 12 octubre). 21. Polinomio de interpolación de Hermite [Vídeo]. YouTube. https://www.youtube.com/watch?v=H57JvLEu8TY&feature=youtu.be_ <br/>
_Süli, Endre & Mayers, David (2003). Hermite Interpolation en An Introduction to Numerical Analysis, primera edición (pp. 187-191). Cambridge University Pres_


Table of Contents

1  Regression

1.0.1  Bibliography: Chapter 8 Ivezic+

2  Regression:

3  Metrics:

3.1  Mean squared error (MSE)

3.2  $MSE \equiv \frac{1}{n} \sum_i (y_i - \hat{y}_i)^2$

3.3  Coefficient of determination (R$^2$)

3.4  $R^2 \equiv 1  - \frac{SS_{res}}{SS_{tot}}$,

3.4.1  $ SS_{res} = \sum_i (y_i - \hat{y}_i)^2$

3.4.2  $ SS_{tot} = \sum_i (y_i - \bar{y})^2$

3.5  Variance and bias

3.6  $Var_{\hat{\theta}} = E_{\hat{\theta}}\biggl[ \bigl(\hat{\theta} - E_{\hat{\theta}}[\hat{\theta}]\bigr)^2\biggr]$

3.7  $Bias_{\hat{\theta}} = E_{\hat{\theta}} - \theta$

3.8  Relation between MSE, variance and bias

4  Linear regression

4.1  Least squares regression for linear regression

4.2  1. Ordinary least squares

4.2.1  $\min RSS = \min \sum\limits_{i=1}^{n} (y_i - \theta_0 - \theta_1 x_i)^2$

4.3  $\hat{\theta}_{1, OLS} = \frac{\sum\limits_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})}{\sum\limits_{i=1}^n (x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}}$

4.4  $\hat{\theta}_{0, OLS} = \bar{y} - \hat{\theta}_{1,OLS} ~\bar{x}$

4.5  $\hat{\sigma}^2 = S^2 = \frac{RSS}{n - 2}$

4.6  $\hat{\theta}_{1, OLS} \sim N\biggl(\theta_1, \frac{\sigma^2}{S_{XX}}\biggr)$

4.7  $\hat{\theta}_{0, OLS} \sim N\biggl(\theta_0, \sigma^2 \biggl(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\biggr)\biggr)$

4.8  $\hat{\sigma}^2 \sim \biggl(\frac{\sigma^2}{n - 2} \biggr) \chi_{n-2}^{2}$

4.9  $ T = \frac{\hat{\theta_1} - \theta_1}{SE}$

4.10  $y(x) = \hat{\theta}_0 + \hat{\theta}_1 x \pm t_{\alpha/2, n - 2} ~ S ~ \sqrt{1 + \frac{1}{n} \frac{(x - \bar{x})^2}{S_{xx}}}$

4.11  2. Maximum likelihood estimator (MLE)

4.12  $p(\lbrace y_i \rbrace | \lbrace x_i \rbrace, \theta) = \prod\limits_{i = 1}^{N} \frac{1}{\sqrt{2 \pi} \sigma_i} \exp \biggl( \frac{- (y_i - (\theta_0 + \theta_1 x_i))^2}{2 \sigma_i^2} \biggr)$

4.13  $\log(L) = \log p(\lbrace y_i \rbrace | \lbrace x_i \rbrace, \theta) = \sum\limits_{i = 1}^{N}  \bigl( \frac{- (y_i - (\theta_0 + \theta_1 x_i))^2}{2 \sigma_i^2} \bigr)$

4.14  $P(\theta | \lbrace y_i \rbrace, \lbrace x_i \rbrace) = P(\lbrace y_i \rbrace | \lbrace x_i \rbrace, \theta) ~ \frac{p(\theta)}{p(y_i)}$

4.15  Online estimation

4.15.1  Diabetes dataset example

4.16  3. Least squares in matrix form

4.17  $y = {\bf M} ~ \theta$

4.18  $y = \begin{bmatrix} y_0 \\ y_1 \\ . \\ y_{N-1} \end{bmatrix}$

4.19  $\theta = \begin{bmatrix} \theta_0 \\ \theta_1 \end{bmatrix}$

4.20  ${\bf M} = \begin{bmatrix} 1 & x_0 \\ 1 & x_1 \\ ... & ... \\ 1 & x_{N-1} \\ \end{bmatrix}$

4.21  $ C = \begin{bmatrix} \sigma_0^2 & 0 & . & 0 \\ 0 & \sigma_1^2 & . & 0 \\ . & . & . & . \\ 0 & 0 & . & \sigma_{N - 1}^2\\ \end{bmatrix}$

4.22  ${\bf \hat{\theta}} = ({\bf M}^T C^{-1} {\bf M})^{-1} ({\bf M}^T C^{-1} {\bf y})$

4.23  $({\bf y} - {\bf M} ~ \theta)^{-1} C^{-1} ({\bf y} -  {\bf M} ~ \theta)$

<a href="#$\Sigma_\theta-=-\begin{bmatrix}-\sigma_{\theta_0}^2-&-\dots-&-\sigma_{\theta_0-\theta_i}-&-\dots-\\-\vdots-&-\ddots-&-\vdots-&-\vdots-\\-\sigma_{\theta_0-\theta_i}-&-\dots-&-\sigma_{\theta_i}^2-\\--\vdots-&-\vdots-&-\vdots-&-\ddots-\\-\end{bmatrix}-=-[{\bf-M}^T-C^{-1}-{\bf-M}]^{-1}$" data-toc-modified-id="$\Sigma_\theta-=-\begin{bmatrix}-\sigma_{\theta_0}^2-&-\dots-&-\sigma_{\theta_0-\theta_i}-&-\dots-\\-\vdots-&-\ddots-&-\vdots-&-\vdots-\\-\sigma_{\theta_0-\theta_i}-&-\dots-&-\sigma_{\theta_i}^2-\\--\vdots-&-\vdots-&-\vdots-&-\ddots-\\-\end{bmatrix}-=-[{\bf-M}^T-C^{-1}-{\bf-M}]^{-1}$-4.24">4.24  $\Sigma_\theta = \begin{bmatrix} \sigma_{\theta_0}^2 & \dots & \sigma_{\theta_0 \theta_i} & \dots \\ \vdots & \ddots & \vdots & \vdots \\ \sigma_{\theta_0 \theta_i} & \dots & \sigma_{\theta_i}^2 \\  \vdots & \vdots & \vdots & \ddots \\ \end{bmatrix} = [{\bf M}^T C^{-1} {\bf M}]^{-1}$

4.25  4. Polynomial regression

4.26  $y_i = \theta_0 + \theta_1 x_i + \theta_2 x_i^2 + \theta_3 x_i^3$

4.27  $ M = \begin{bmatrix}
1 & x_0 & x_0^2 & x_0^3 \\
1 & x_1 & x_1^2 & x_1^3 \\
. & . & . & . \\
1 & x_N & x_N^2 & x_N^3 \\
\end{bmatrix}$

5  Regularization

5.1  1. Ridge regression

5.2  $({\bf y} - M~ \theta)^{-1} C^{-1}({\bf y} - M~ \theta) + \lambda~ |\theta^T \theta|$

5.3  ${\bf \hat{\theta}} = ({\bf M}^T C^{-1} {\bf M} + \lambda I)^{-1} ({\bf M}^T C^{-1} {\bf y})$

5.3.1  Bayes equivalent

5.4  $ p(\theta) \propto \exp \biggl( \frac{- \lambda (\theta^T \theta)}{2} \biggr)$

<a href="#$|{\bf-\theta}|^2-<-s$" data-toc-modified-id="$|{\bf-\theta}|^2-<-s$-5.5">5.5  $|{\bf \theta}|^2 < s$

5.6  2. LASSO regression

5.7  $({\bf y} - \theta {\bf X})^{-1}({\bf y} - \theta {\bf X}) + \lambda |\theta|$

<a href="#$|{\bf-\theta}|-<-s$" data-toc-modified-id="$|{\bf-\theta}|-<-s$-5.8">5.8  $|{\bf \theta}| < s$

5.8.1  Understanding Ridge and LASSO regression

6  Comparison between ordinary, Ridge and LASSO regression

6.1  Choosing the regularization parameter $\lambda$

6.2  $ Err(\lambda) = k^{1} \sum_k N_k^{-1} \sum\limits_{i}^{N_k} \frac{[y_i - f(x_i | \theta)]^2}{\sigma_i^2}$

7  Outliers

7.1  Huber loss

7.1.1  $ \sum\limits_{i=1}^N e(y_i|y)$

7.1.2  $\begin{equation}
  \phi(t)=\left\{
  \begin{array}{@{}ll@{}}
    \frac{1}{2} t^2, & \text{if}\ |t|\le c \\
    c|t| - \frac{1}{2} c^2, & \text{otherwise}
  \end{array}\right.
\end{equation}$

7.2  Bayesian Outlier Methods

8  Cross-Validation Testing

```python
%config InlineBackend.figure_format = 'retina'
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

plt.style.use(['ggplot', 'assets/class.mplstyle'])

red = '#E24A33'
blue = '#348ABD'
purple = '#988ED5'
gray = '#777777'
yellow = '#FBC15E'
green = '#8EBA42'
pink = '#FFB5B8'
```

# Regression

### Bibliography: Chapter 8 Ivezic+

# Regression:

- Finding the relation between a dependent variable $y$ and a set of independent variables $\bf x$. It describes the expectation value of $y$ given $\bf x$, $E[y|{\bf x}]$.

- The dependent variable $y$ is also called **response variable**.

- The problem is to find the expectation of $y$ given $\bf x$:

$E[y|{\bf x}] = f({\bf x}, \theta) + \epsilon$

where $\epsilon$ is a random error.

Note that we estimate $E[y | {\bf x}]$, the conditional expectation of $y$ given $\bf x$, rather than $p(y, {\bf x})$, the joint distribution of $y$ and $\bf x$, which is a much harder problem to solve. Regression is a simpler problem: instead of determining the multidimensional pdf, we would like to infer the expectation of $y$ given $\bf x$.

**The goal of regression usually focuses on estimation of the parameters $\theta$ of the function and their confidence intervals.**

# Metrics:

## Mean squared error (MSE)

The mean squared error (MSE) or mean squared deviation (MSD) of an estimator measures the average of the squares of the errors:

It is defined by (estimation of $y$):

## $MSE \equiv \frac{1}{n} \sum_i (y_i - \hat{y}_i)^2$

## Coefficient of determination (R$^2$)

It is the proportion of the variance in the dependent variable that is predictable from the independent variable(s).

It is defined as:

## $R^2 \equiv 1  - \frac{SS_{res}}{SS_{tot}}$,

where

### $ SS_{res} = \sum_i (y_i - \hat{y}_i)^2$

and

### $ SS_{tot} = \sum_i (y_i - \bar{y})^2$

## Variance and bias

The **variance** of an estimator of the parameter $\theta$ will be:

## $Var_{\hat{\theta}} = E_{\hat{\theta}}\biggl[ \bigl(\hat{\theta} - E_{\hat{\theta}}[\hat{\theta}]\bigr)^2\biggr]$

The **bias** of an estimator will be:

## $Bias_{\hat{\theta}} = E_{\hat{\theta}} - \theta$

## Relation between MSE, variance and bias

Note that when estimating a parameter the MSE equals the variance plus the bias squared

# Linear regression

The one dimensional linear regression model is defined by

$f(x, \theta) = \theta_0 + \theta_1 x$

## Least squares regression for linear regression

## 1. Ordinary least squares

Minimizes the residual sum of squares (RSS) between the observed response values $y_i$ and the model predictions

### $\min RSS = \min \sum\limits_{i=1}^{n} (y_i - \theta_0 - \theta_1 x_i)^2$

Differentiating w.r.t. to $\theta_0$ and $\theta_1$ and setting these derivatives to zero gives two equations which can be solved to give:

## $\hat{\theta}_{1, OLS} = \frac{\sum\limits_{i=1}^n (x_i - \bar{x}) (y_i - \bar{y})}{\sum\limits_{i=1}^n (x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}}$

## $\hat{\theta}_{0, OLS} = \bar{y} - \hat{\theta}_{1,OLS} ~\bar{x}$

OLS = ordinary least squares

These estimators are also the maximum likelihood estimator (MLE) when the errors are normally distributed.

The OLS estimator of the error variance $\sigma^2$ is:

## $\hat{\sigma}^2 = S^2 = \frac{RSS}{n - 2}$

If the errors are i.i.d. random variables, the slope and intercept are unbiased and asymptotically normally distributed:

## $\hat{\theta}_{1, OLS} \sim N\biggl(\theta_1, \frac{\sigma^2}{S_{XX}}\biggr)$

## $\hat{\theta}_{0, OLS} \sim N\biggl(\theta_0, \sigma^2 \biggl(\frac{1}{n} + \frac{\bar{x}^2}{S_{XX}}\biggr)\biggr)$

If the errors are also normally distributed with zero mean and known variance $\sigma^2$, then the variance is $\chi^2$ distributed:

## $\hat{\sigma}^2 \sim \biggl(\frac{\sigma^2}{n - 2} \biggr) \chi_{n-2}^{2}$

However, $\sigma^2$ is rarely known, and only the sample variance $S^2$ is known, changing the statistical properties of the estimators.

Defining:

## $ T = \frac{\hat{\theta_1} - \theta_1}{SE}$

where $SE = \frac{S}{\sqrt{S_{XX}}}$ is also known as the **standard error**,

one can show that $T$ is t (t-Student) distributed with $n-2$ degrees of freedom.

Then, the $(1-\alpha)$ confidence interval for $\theta_1$ becomes $(\hat{\theta}_1 - t_{\alpha/2, n-2} ~ SE, \hat{\theta}_1 + t_{\alpha/2, n-2} ~ SE)$ and the confidence interval for $y$ at a given $x$ becomes:

## $y(x) = \hat{\theta}_0 + \hat{\theta}_1 x \pm t_{\alpha/2, n - 2} ~ S ~ \sqrt{1 + \frac{1}{n} \frac{(x - \bar{x})^2}{S_{xx}}}$

## 2. Maximum likelihood estimator (MLE)

Let us consider the probability of observing $\lbrace y_i \rbrace$ given the parameters $\theta$ and the set of independent variable $\lbrace x_i \rbrace$.

Here we assume that the observations are Gaussian with standard deviation $\sigma_i$:

## $p(\lbrace y_i \rbrace | \lbrace x_i \rbrace, \theta) = \prod\limits_{i = 1}^{N} \frac{1}{\sqrt{2 \pi} \sigma_i} \exp \biggl( \frac{- (y_i - (\theta_0 + \theta_1 x_i))^2}{2 \sigma_i^2} \biggr)$

Thus, we can maximize the log likelihood minus a constant to find the **maximum likelihood estimator (MLE)**

## $\log(L) = \log p(\lbrace y_i \rbrace | \lbrace x_i \rbrace, \theta) = \sum\limits_{i = 1}^{N}  \bigl( \frac{- (y_i - (\theta_0 + \theta_1 x_i))^2}{2 \sigma_i^2} \bigr)$

which is **equivalent to minimizing the RSS**

Using Bayes theorem $P(A|B) = P(B|A)~\dfrac{P(A)}{P(B)}$ we can also see that

## $P(\theta | \lbrace y_i \rbrace, \lbrace x_i \rbrace) = P(\lbrace y_i \rbrace | \lbrace x_i \rbrace, \theta) ~ \frac{p(\theta)}{p(y_i)}$

Therefore, this is also the **maximum a posteriori (MAP)** estimator assuming a flat prior for $\theta$.

## Online estimation

This figure illustrates how the uncertainties in the parameters shrink as the number of observations grow.

### Diabetes dataset example
(https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html#sphx-glr-auto-examples-linear-model-plot-ols-py)

We want to try and figure out how bmi affects how sick the person is after one year.

```python
import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
from sklearn.metrics import mean_squared_error, r2_score

# Load the diabetes dataset
diabetes = datasets.load_diabetes()
print(diabetes.DESCR)
```

    .. _diabetes_dataset:

    Diabetes dataset
    ----------------

    Ten baseline variables, age, sex, body mass index, average blood
    pressure, and six blood serum measurements were obtained for each of n =
    442 diabetes patients, as well as the response of interest, a
    quantitative measure of disease progression one year after baseline.

    **Data Set Characteristics:**

      :Number of Instances: 442

      :Number of Attributes: First 10 columns are numeric predictive values

      :Target: Column 11 is a quantitative measure of disease progression one year after baseline

      :Attribute Information:
          - age     age in years
          - sex
          - bmi     body mass index
          - bp      average blood pressure
          - s1      tc, total serum cholesterol
          - s2      ldl, low-density lipoproteins
          - s3      hdl, high-density lipoproteins
          - s4      tch, total cholesterol / HDL
          - s5      ltg, possibly log of serum triglycerides level
          - s6      glu, blood sugar level

    Note: Each of these 10 feature variables have been mean centered and scaled by the standard deviation times `n_samples` (i.e. the sum of squares of each column totals 1).

    Source URL:
    https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html

    For more information see:
    Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani (2004) "Least Angle Regression," Annals of Statistics (with discussion), 407-499.
    (https://web.stanford.edu/~hastie/Papers/LARS/LeastAngle_2002.pdf)

```python
# Use only one feature (body mass index)
diabetes_X = diabetes.data[:, np.newaxis, 2]  # axis 2 = bmi

# Split the data into training/testing sets
diabetes_X_train = diabetes_X[:-20]  # last 20 points
diabetes_X_test = diabetes_X[-20:]  # we put them into the test set

# Split the targets into training/testing sets accordingly
diabetes_y_train = diabetes.target[:-20]
diabetes_y_test = diabetes.target[-20:]

# Create linear regression object
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(diabetes_X_train, diabetes_y_train)

# Make predictions using the testing set
diabetes_y_pred = regr.predict(diabetes_X_test)

# The coefficients
print('Coefficients: \n', regr.coef_)
# The mean squared error
print("Mean squared error (MSE): %.2f"
      % mean_squared_error(diabetes_y_test, diabetes_y_pred))
# Explained variance score: 1 is perfect prediction
print('Variance score (coefficient of determination, R2): %.2f' % r2_score(diabetes_y_test, diabetes_y_pred))
```

    Coefficients:
     [938.23786125]
    Mean squared error (MSE): 2548.07
    Variance score (coefficient of determination, R2): 0.47

```python
fig, ax = plt.subplots(figsize = (11, 7))

ax.scatter(diabetes_X_test, diabetes_y_test,  c='k')
ax.scatter(diabetes_X_train, diabetes_y_train, c='gray', alpha=0.3)
ax.plot(diabetes_X_test, diabetes_y_pred, color=blue, linewidth=3)
ax.set_xlabel("BMI (scaled)")
ax.set_ylabel("Diabetes disease progression")
```

## 3. Least squares in matrix form

The previous equations can be derived in matrix form

In general one can define regression in terms of a design matrix $M$:

## $y = {\bf M} ~ \theta$

where $y$ is the $N$–dimensional vector of values $y_i$.

## $y = \begin{bmatrix} y_0 \\ y_1 \\ . \\ y_{N-1} \end{bmatrix}$

For straight--line regression, $\theta$ is a two--dimensional vector:

## $\theta = \begin{bmatrix} \theta_0 \\ \theta_1 \end{bmatrix}$

and $M$ is a $2 \times N$ matrix:

## ${\bf M} = \begin{bmatrix} 1 & x_0 \\ 1 & x_1 \\ ... & ... \\ 1 & x_{N-1} \\ \end{bmatrix}$

For the case of heteroscedastic (different) uncertainties, we can define the covariance matrix $C$:

## $ C = \begin{bmatrix} \sigma_0^2 & 0 & . & 0 \\ 0 & \sigma_1^2 & . & 0 \\ . & . & . & . \\ 0 & 0 & . & \sigma_{N - 1}^2\\ \end{bmatrix}$

The MLE solution for this regression is:

## ${\bf \hat{\theta}} = ({\bf M}^T C^{-1} {\bf M})^{-1} ({\bf M}^T C^{-1} {\bf y})$

which is the also the solution that minimizes the equation

## $({\bf y} - {\bf M} ~ \theta)^{-1} C^{-1} ({\bf y} -  {\bf M} ~ \theta)$

The uncertainties on the regression coefficients $\bf \theta$ can be expressed as the following matrix (general case):

## $\Sigma_\theta = \begin{bmatrix} \sigma_{\theta_0}^2 & \dots & \sigma_{\theta_0 \theta_i} & \dots \\ \vdots & \ddots & \vdots & \vdots \\ \sigma_{\theta_0 \theta_i} & \dots & \sigma_{\theta_i}^2 \\  \vdots & \vdots & \vdots & \ddots \\ \end{bmatrix} = [{\bf M}^T C^{-1} {\bf M}]^{-1}$

Values outside of the diagonal appear when they are not fully independent.

In the general case, $C$ can be non--diagonal too, which indicates correlated noise.

We can extend it to...

## 4. Polynomial regression

Let us consider the case where we perform a polynomial representation of the data, e.g.:

## $y_i = \theta_0 + \theta_1 x_i + \theta_2 x_i^2 + \theta_3 x_i^3$

In this case the design matrix $M$ becomes:

## $ M = \begin{bmatrix}
1 & x_0 & x_0^2 & x_0^3 \\
1 & x_1 & x_1^2 & x_1^3 \\
. & . & . & . \\
1 & x_N & x_N^2 & x_N^3 \\
\end{bmatrix}$

And the previous formulae can be used to find the least squares solution since the coefficients are linear

# Regularization

Sometimes we may want to trade an increase in bias for a reduction in variance (e.g. if the matrix $M$ is ill-conditioned the variance could be very large)

## 1. Ridge regression

In this case we include the following regularization term in the least squares equation:

## $({\bf y} - M~ \theta)^{-1} C^{-1}({\bf y} - M~ \theta) + \lambda~ |\theta^T \theta|$

$\lambda$ is called the **regularization or smoothing parameter** and **$|\theta^T \theta|$** is called the **penalty function**

It can be shown that the new solution will be:

## ${\bf \hat{\theta}} = ({\bf M}^T C^{-1} {\bf M} + \lambda I)^{-1} ({\bf M}^T C^{-1} {\bf y})$

It is worth noting that even if $M^T C^{-1} M$ were singular, $M^T C^{-1} M + \lambda I$ may not be singular

=> $\lambda$ wants $\theta$ to have elements with less variance

"Continuous"

### Bayes equivalent

In the Bayesian approach, it can be shown that the MAP solution is the same as the ridge solution if we assume a Gaussian prior for the parameters $\theta$:

## $ p(\theta) \propto \exp \biggl( \frac{- \lambda (\theta^T \theta)}{2} \biggr)$

Ridge regression effectively constrains the norm of the parameters $\theta$:

## $|{\bf \theta}|^2 < s$

## 2. LASSO regression

Another common regularization technique

In this case we include a different regularization term in the least squares equation:

## $({\bf y} - \theta {\bf X})^{-1}({\bf y} - \theta {\bf X}) + \lambda |\theta|$

Now we use the $\ell_1$ norm instead of the $\ell_2$ norm.

In this case there is no closed form, but it can be shown that the MAP solution is the same if an exponential prior for the parameters $\theta$ is assumed.

LASSO regression effectively constrains the $\ell_1$ norm of the parameters $\theta$:

## $|{\bf \theta}| < s$

=> $\lambda$ wants $\theta$ to have less elements

"Discrete"

### Understanding Ridge and LASSO regression

# Comparison between ordinary, Ridge and LASSO regression

```python
from sklearn import linear_model

X_train = np.c_[.5, 1].T
y_train = [.5, 1]
X_test = np.c_[0, 2].T

np.random.seed(0)

classifiers = dict(ols=linear_model.LinearRegression(),
                   ridge=linear_model.Ridge(alpha=.05),
                    lasso=linear_model.Lasso(alpha=.02))

for name, clf in classifiers.items():
    fig, ax = plt.subplots(figsize=(4, 3))

    for _ in range(6):
        this_X = .1 * np.random.normal(size=(2, 1)) + X_train
        clf.fit(this_X, y_train)

        ax.plot(X_test, clf.predict(X_test), color='gray')
        ax.scatter(this_X, y_train, s=3, c='gray', marker='o', zorder=10)

    clf.fit(X_train, y_train)
    ax.plot(X_test, clf.predict(X_test), linewidth=2, color=blue)
    ax.scatter(X_train, y_train, s=30, c=red, marker='+', zorder=10)

    ax.set_title(name)
    ax.set_xlim(0, 2)
    ax.set_ylim((0, 1.6))
    ax.set_xlabel('X')
    ax.set_ylabel('y')

    fig.tight_layout()

plt.show()
```

## Choosing the regularization parameter $\lambda$

One way to choose $\lambda$ is to do k-fold cross-validation on $\lambda$, defining the error:

## $ Err(\lambda) = k^{1} \sum_k N_k^{-1} \sum\limits_{i}^{N_k} \frac{[y_i - f(x_i | \theta)]^2}{\sigma_i^2}$

and then choose the $\lambda$ with the smallest error.

# Outliers
## Huber loss

The Huber estimator minimizes

### $ \sum\limits_{i=1}^N e(y_i|y)$

where $e(y_i|y)$ is modeled as:

### $\begin{equation}
  \phi(t)=\left\{
  \begin{array}{@{}ll@{}}
    \frac{1}{2} t^2, & \text{if}\ |t|\le c \\
    c|t| - \frac{1}{2} c^2, & \text{otherwise}
  \end{array}\right.
\end{equation}$

With $t = y_i - \hat{y}_i$

c is a new parameter

If it's close to the line, it is treated as a Gaussian. If it is far, as an exponencial.

## Bayesian Outlier Methods

Alternative to the Huber loss, we can modify the likelihood function by assuming a mixture model, where some points belong to an outlier distribution, which is modeled as a Gaussian with mean $\mu_b$ and variance $V_b$, and others belong to the main population, which follow the linear model.

1. Assume that the probability of belonging to the outlier population is $p_b$

2. Assume that each point can be tagged as either outlier ($g_i = 0$) or not ($g_i = 1$)

```python

```

```python
from sklearn.datasets import make_regression
from sklearn.linear_model import HuberRegressor, Ridge

# Generate toy data.
rng = np.random.RandomState(0)
X, y = make_regression(n_samples=20, n_features=1, random_state=0, noise=4.0,
                       bias=100.0)

# Add four strong outliers to the dataset.
X_outliers = rng.normal(0, 0.5, size=(4, 1))
y_outliers = rng.normal(0, 2.0, size=4)
X_outliers[:2, :] += X.max() + X.mean() / 4.
X_outliers[2:, :] += X.min() - X.mean() / 4.
y_outliers[:2] += y.min() - y.mean() / 4.
y_outliers[2:] += y.max() + y.mean() / 4.
X = np.vstack((X, X_outliers))
y = np.concatenate((y, y_outliers))
plt.plot(X, y, 'b.')

# Fit the huber regressor over a series of epsilon values.
colors = ['r-', 'b-', 'y-', 'm-']

x = np.linspace(X.min(), X.max(), 7)
epsilon_values = [1.35, 1.5, 1.75, 1.9]
for k, epsilon in enumerate(epsilon_values):
    huber = HuberRegressor(fit_intercept=True, alpha=0.0, max_iter=100,
                           epsilon=epsilon)
    huber.fit(X, y)
    coef_ = huber.coef_ * x + huber.intercept_
    plt.plot(x, coef_, colors[k], label="huber loss, %s" % epsilon)

# Fit a ridge regressor to compare it to huber regressor.
ridge = Ridge(fit_intercept=True, alpha=0.0, random_state=0, normalize=True)
ridge.fit(X, y)
coef_ridge = ridge.coef_
coef_ = ridge.coef_ * x + ridge.intercept_
plt.plot(x, coef_, 'g-', label="ridge regression")

plt.title("Comparison of HuberRegressor vs Ridge")
plt.xlabel("X")
plt.ylabel("y")
plt.legend(loc=0)
plt.show()
```

# Cross-Validation Testing

Analogous to the classification problem, one can also do cross-validation during regression to determine the best hyperparameters. For example, choosing the regularization parameter for ridge or lasso regression, or choosing the polynomial degree of a polynomial fit.

=> **how to know when to stop increasing the degree of the polynomial?**

We want to get close to the data points, but not EXACTLY. We want the underlying distribution, not get this specific set.

We want to be somewhere in the good range that makes it better for both sets.

```python

```


Demo - Poisson's equation 1D
=======================

In this demo we will solve Poisson's equation

\begin{align}
\label{eq:poisson}
\nabla^2 u(x) &= f(x), \quad \forall \, x \in [-1, 1]\\
u(\pm 1) &= 0,
\end{align}

where $u(x)$ is the solution and $f(x)$ is some given function of $x$.

We want to solve this equation with the spectral Galerkin method, using a basis composed of either Chebyshev $T_k(x)$ or Legendre $L_k(x)$ polynomials. Using $P_k$ to refer to either one, Shen's composite basis is then given as

$$
V^N = \text{span}\{P_k - P_{k+2}\, | \, k=0, 1, \ldots, N-3\},
$$

where all basis functions satisfy the homogeneous boundary conditions.

For the spectral Galerkin method we will also need the weighted inner product

$$
 (u, v)_w = \int_{-1}^1 u v w \, {dx},
$$

where $w(x)$ is a weight associated with the chosen basis, and $v$ and $u$ are test and trial functions, respectively. For Legendre the weight is simply $w(x)=1$, whereas for Chebyshev it is $w(x)=1/\sqrt{1-x^2}$. Quadrature is used to approximate the integral

$$
\int_{-1}^1 u v w \, {dx} \approx \sum_{i=0}^{N-1} u(x_i) v(x_i) \omega_i,
$$

where $\{\omega_i\}_{i=0}^{N-1}$ are the quadrature weights associated with the chosen basis and quadrature rule. The associated quadrature points are denoted as $\{x_i\}_{i=0}^{N-1}$. For Chebyshev we can choose between `Chebyshev-Gauss` or `Chebyshev-Gauss-Lobatto`, whereas for Legendre the choices are `Legendre-Gauss` or `Legendre-Gauss-Lobatto`.

With the weighted inner product in place we can create variational problems from the original PDE by multiplying with a test function $v$ and integrating over the domain. For a Legendre basis we can use integration by parts and formulate the variational problem:

Find $u \in V^N$ such that

$$     (\nabla u, \nabla v) = -(f, v), \quad \forall \, v \in V^N.$$

For a Chebyshev basis the integration by parts is complicated due to the non-constant weight and the variational problem used is instead:

Find $u \in V^N$ such that

$$     (\nabla^2 u, v)_w = (f, v)_w, \quad \forall \, v \in V^N.$$

We now break the problem down to linear algebra. With any choice of basis or quadrature rule we use $\phi_k(x)$ to represent the test function $v$ and thus

$$
\begin{align}
v(x) &= \phi_k(x), \\
u(x) &= \sum_{j=0}^{N-3} \hat{u}_j \phi_j(x),
\end{align}
$$
where $\hat{\mathbf{u}}=\{\hat{u}_j\}_{j=0}^{N-3}$ are the unknown expansion coefficients, also called the degrees of freedom.

Insert into the variational problem for Legendre and we get the linear algebra system to solve for $\hat{\mathbf{u}}$

$$
\begin{align}
(\nabla \sum_{j=0}^{N-3} \hat{u}_j \phi_j, \nabla \phi_k) &= -(f, \phi_k), \\
\sum_{j=0}^{N-3} \underbrace{(\nabla \phi_j, \nabla \phi_k)}_{a_{kj}} \hat{u}_j  &= -\underbrace{(f, \phi_k)}_{\tilde{f}_k}, \\
A \hat{\textbf{u}} &= -\tilde{\textbf{f}},
\end{align}
$$

where $A = (a_{kj})_{0 \ge k, j \ge N-3}$ is the stiffness matrix and $\tilde{\textbf{f}} = \{\tilde{f}_k\}_{k=0}^{N-3}$.

## Implementation with shenfun

The given problem may be easily solved with a few lines of code using the shenfun Python module. The high-level code matches closely the mathematics and the stiffness matrix is assembled simply as

```python
from shenfun import *
import matplotlib.pyplot as plt

N = 100
V = FunctionSpace(N, 'Legendre', quad='LG', bc=(0, 0))
v = TestFunction(V)
u = TrialFunction(V)
A = inner(grad(u), grad(v))

```

Using a manufactured solution that satisfies the boundary conditions we can create just about any corresponding right hand side $f(x)$

```python
import sympy
x = sympy.symbols('x')
ue = (1-x**2)*(sympy.cos(4*x)*sympy.sin(6*x))
fe = ue.diff(x, 2)
```

Note that `fe` is the right hand side that corresponds to the exact solution `ue`. We now want to use `fe` to compute a numerical solution $u$ that can be compared directly with the given `ue`. First, to compute the inner product $(f, v)$, we need to evaluate `fe` on the quadrature mesh

```python
fl = sympy.lambdify(x, fe, 'numpy')
ul = sympy.lambdify(x, ue, 'numpy')
fj = Array(V, buffer=fl(V.mesh()))
```

`fj` holds the analytical `fe` on the nodes of the quadrature mesh.
Assemble right hand side $\tilde{\textbf{f}} = -(f, v)_w$ using the shenfun function `inner`

```python
f_tilde = inner(-fj, v)
```

All that remains is to solve the linear algebra system

$$
\begin{align}
A \hat{\textbf{u}} &= \tilde{\textbf{f}} \\
\hat{\textbf{u}} &= A^{-1} \tilde{\textbf{f}}
\end{align}
$$

```python
u_hat = Function(V)
u_hat = A/f_tilde

```

Get solution in real space, i.e., evaluate $u(x_i) = \sum_{j=0}^{N-3} \hat{u}_j \phi_j(x_i)$ for all quadrature points $\{x_i\}_{i=0}^{N-1}$.

```python
uj = u_hat.backward()
```

```python
X = V.mesh()
plt.plot(X, uj)
```


```python
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Discussion: Week 2

```python
# Import the NumPy module

```

## Exercise: Capital Evolution in the Solow Model

Suppose that capital per worker $k_t$ evolves according to the following equation:

\begin{align}
k_{t+1} & = 0.12 \cdot 100 \cdot k_t^{1/3} + 0.9\cdot k_t, \tag{1}
\end{align}

where the first term on the right-hand side implies that the economy has a 12 percent savings rate, that total factor productivity equals 100, and that there is no growth in technology (or "labor efficiency"). The second term implies that the rate of capital depreciation is 10 percent (i.e., $1-\delta = 0.9 \Rightarrow \delta = 0.1$). Assume that capital per worker in the initial period $k_0$ is given.

The *steady state* quantity of capital per worker is the number $k^*$ such that if $k_t = k^*$, $k_{t+1} = k^*$. Find $k^*$ by dropping the time subscripts in equation (1) and solving for $k$. Obtain:

\begin{align}
k^* & = \left(\frac{0.1}{0.12\cdot 100}\right)^{3/2} = 1{,}314.53414 \tag{2}
\end{align}

### Part (a): Simulate 100 Periods

```python
# Create a variable called 'k0' that stores the initial quantity of capital in the economy. Set 'k0' to 400

# Create a variable called 'T' equal to the number of periods after 0 to simulate. Set T = 100

# Use the function np.zeros to create a variable called 'capital' equal to an array of zeros of length T+1

# Print the value of 'capital'

```

```python
# Set the first element of 'capital' to the value in k0

# Print the value of 'capital'

```

```python
# Use a for loop to iterate over the additional elemnts of the 'capital' array that need to be computed.
# Hint: capital has length T+1. The first value is filled, so you need fill the remaining T values.

# Print the value of 'capital'

```

```python
# Print the value of  the last element of 'capital'

```

```python
# Plot the simulated capital per worker

```

### Part (b): Simulate 1,000 Periods

```python
# Create a variable called 'T' equal to the number of periods after 0 to simulate. Set T = 1000

# Use the function np.zeros to create a variable called 'capital' equal to an array of zeros of length T+1

# Set the first element of 'capital' to the value in k0

# Use a for loop to iterate over the additional elemnts of the 'capital' array that need to be computed.

# Print the value of  the last element of 'capital'

```

### Part (c): Evaluation

Provide answers to the follow questions in the next cell.

**Question**

1. Why is the final value of capital computed in Part (b) closer to the true steady state than the value computed in Part (a)?

**Answer**

1.


## Example: Hierarchical Gaussian Filter

In this demo the goal is to perform approximate variational Bayesian Inference for Univariate Hierarchical Gaussian Filter (HGF).

Simple HGF model can be defined as:

$$
\begin{equation}
  \begin{aligned}
    x^{(j)}_k & \sim \, \mathcal{N}(x^{(j)}_{k - 1}, f_k(x^{(j - 1)}_k)) \\
    y_k & \sim \, \mathcal{N}(x^{(j)}_k, \tau_k)
  \end{aligned}
\end{equation}
$$

where $j$ is an index of layer in hierarchy, $k$ is a time step and $f_k$ is a variance activation function. `ReactiveMP.jl` export Gaussian Controlled Variance (GCV) node with $f_k = \exp(\kappa x + \omega)$ variance activation function. By default uses Gauss-Hermite cubature with a prespecified number of approximation points in the cubature. We can change the number of points in Gauss-Hermite cubature with the help of metadata structures in `ReactiveMP.jl`.

$$
\begin{equation}
  \begin{aligned}
    z_k & \sim \, \mathcal{N}(z_{k - 1}, \mathcal{\tau_z}) \\
    x_k & \sim \, \mathcal{N}(x_{k - 1}, \exp(\kappa z_k + \omega)) \\
    y_k & \sim \, \mathcal{N}(x_k, \mathcal{\tau_y})
  \end{aligned}
\end{equation}
$$

In this experiment we will create a single time step of the graph and perform variational message passing filtering alrogithm to estimate hidden states of the system. For a more rigorous introduction to Hierarchical Gaussian Filter we refer to [Ismail Senoz, Online Message Passing-based Inference in the Hierarchical Gaussian Filter](https://ieeexplore.ieee.org/document/9173980) paper.

For simplicity we will consider $\tau_z$, $\tau_y$, $\kappa$ and $\omega$ known and fixed.

To model this process in `ReactiveMP`, first, we start with importing all needed packages:

```julia
using Rocket, ReactiveMP, GraphPPL, Distributions
using BenchmarkTools, Random, Plots
```

    ┌ Info: Precompiling ReactiveMP [a194aa59-28ba-4574-a09c-4a745416d6e3]
    └ @ Base loading.jl:1342

Next step, is to generate some synthetic data:

```julia
function generate_data(rng, k, w, zv, yv)
    z_prev = 0.0
    x_prev = 0.0

    z = Vector{Float64}(undef, n)
    v = Vector{Float64}(undef, n)
    x = Vector{Float64}(undef, n)
    y = Vector{Float64}(undef, n)

    for i in 1:n
        z[i] = rand(rng, Normal(z_prev, sqrt(zv)))
        v[i] = exp(k * z[i] + w)
        x[i] = rand(rng, Normal(x_prev, sqrt(v[i])))
        y[i] = rand(rng, Normal(x[i], sqrt(yv)))

        z_prev = z[i]
        x_prev = x[i]
    end

    return z, x, y
end
```

    generate_data (generic function with 1 method)

```julia
# Seed for reproducibility
seed = 123

rng = MersenneTwister(seed)

# Parameters of HGF process
real_k = 1.0
real_w = 0.0
z_variance = abs2(0.5)
y_variance = abs2(1.0)

# Number of observations
n = 300

z, x, y = generate_data(rng, real_k, real_w, z_variance, y_variance);
```

Lets plot our synthetic dataset. Lines represent our hidden states we want to estimate using noisy observations.

```julia
let
    pz = plot(title = "Hidden States Z")
    px = plot(title = "Hidden States X")

    plot!(pz, 1:n, z, label = "z_i", color = :orange)
    plot!(px, 1:n, x, label = "x_i", color = :green)

    plot(pz, px, layout = @layout([ a; b ]))
end
```

To create a model we use `GraphPPL` package and `@model` macro:

```julia
# We create a single-time step of corresponding state-space process to
# perform online learning (filtering)
@model [ default_factorisation = MeanField() ] function hgf(real_k, real_w, z_variance, y_variance)

    # Priors from previous time step for `z`
    zt_min_mean = datavar(Float64)
    zt_min_var  = datavar(Float64)

    # Priors from previous time step for `x`
    xt_min_mean = datavar(Float64)
    xt_min_var  = datavar(Float64)

    zt_min ~ NormalMeanVariance(zt_min_mean, zt_min_var)
    xt_min ~ NormalMeanVariance(xt_min_mean, xt_min_var)

    # Higher layer is modelled as a random walk
    zt ~ NormalMeanVariance(zt_min, z_variance) where { q = q(zt, zt_min)q(z_variance) }

    # Lower layer is modelled with `GCV` node
    gcv_node, xt ~ GCV(xt_min, zt, real_k, real_w) where { q = q(xt, xt_min)q(zt)q(κ)q(ω) }

    # Noisy observations
    y = datavar(Float64)
    y ~ NormalMeanVariance(xt, y_variance)

    return zt, xt, y, gcv_node, xt_min_mean, xt_min_var, zt_min_mean, zt_min_var
end
```

    hgf (generic function with 1 method)

```julia
function inference(data, vmp_iters, real_k, real_w, z_variance, y_variance)
    n = length(data)

    # We don't want to save all marginals from all VMP iterations
    # but only last one after all VMP iterations per time step
    # Rocket.jl exports PendingScheduler() object that postpones
    # any update unless manual `resolve!()` has been called
    ms_scheduler = PendingScheduler()

    mz = keep(Marginal)
    mx = keep(Marginal)
    fe = ScoreActor(Float64)

    model, (zt, xt, y, gcv_node, xt_min_mean, xt_min_var, zt_min_mean, zt_min_var) = hgf(real_k, real_w, z_variance, y_variance)

    # Initial priors
    current_zt_mean, current_zt_var = 0.0, 10.0
    current_xt_mean, current_xt_var = 0.0, 10.0

    s_mz = subscribe!(getmarginal(zt) |> schedule_on(ms_scheduler), mz)
    s_mx = subscribe!(getmarginal(xt) |> schedule_on(ms_scheduler), mx)
    s_fe = subscribe!(score(Float64, BetheFreeEnergy(), model), fe)

    # Initial marginals to start VMP procedire
    setmarginal!(gcv_node, :y_x, MvNormalMeanCovariance([ 0.0, 0.0 ], [ 5.0, 5.0 ]))
    setmarginal!(gcv_node, :z, NormalMeanVariance(0.0, 5.0))

    # For each observations we perofrm `vmp_iters` VMP iterations
    for i in 1:n

        for _ in 1:vmp_iters
            update!(y, data[i])
            update!(zt_min_mean, current_zt_mean)
            update!(zt_min_var, current_zt_var)
            update!(xt_min_mean, current_xt_mean)
            update!(xt_min_var, current_xt_var)
        end

        # After all VMP iterations we release! `PendingScheduler`
        # as well as release! `ScoreActor` to indicate new time step
        release!(ms_scheduler)
        release!(fe)

        current_zt_mean, current_zt_var = mean_var(last(mz))::Tuple{Float64, Float64}
        current_xt_mean, current_xt_var = mean_var(last(mx))::Tuple{Float64, Float64}
    end

    # It is important to unsubscribe at the end of the inference procedure
    unsubscribe!((s_mz, s_mx, s_fe))

    return map(getvalues, (mz, mx, fe))
end
```

    inference (generic function with 1 method)

To run inference we also specify number of VMP iterations we want to perform as well as an approximation method for GCV node:

```julia
vmp_iters = 10
```

    10

```julia
mz, mx, fe = inference(y, vmp_iters, real_k, real_w, z_variance, y_variance);
```

```julia
let
    pz = plot(title = "Hidden States Z")
    px = plot(title = "Hidden States X")

    plot!(pz, 1:n, z, label = "z_i", color = :orange)
    plot!(pz, 1:n, mean.(mz), ribbon = std.(mz), label = "estimated z_i", color = :teal)

    plot!(px, 1:n, x, label = "x_i", color = :green)
    plot!(px, 1:n, mean.(mx), ribbon = std.(mx), label = "estimated x_i", color = :violet)

    plot(pz, px, layout = @layout([ a; b ]))
end
```

As we can see from our plot, estimated signal resembles closely to the real hidden states with small variance. We maybe also interested in the values for Bethe Free Energy functional:

```julia
plot(fe, label = "Bethe Free Energy")
```

As we can see BetheFreeEnergy converges nicely to a stable point.

We may be also interested in performance of our resulting Variational Message Passing algorithm:

```julia
@benchmark inference($y, $vmp_iters, $real_k, $real_w, $z_variance, $y_variance)
```

    BenchmarkTools.Trial: 59 samples with 1 evaluation.
     Range (min … max):  73.028 ms … 92.583 ms  ┊ GC (min … max):  0.00% … 10.27%
     Time  (median):     86.755 ms              ┊ GC (median):    10.30%
     Time  (mean ± σ):   84.754 ms ±  5.866 ms  ┊ GC (mean ± σ):   6.92% ±  4.99%

              ▁         ▁       ▄      ▁        ▄▁ ▄  ▁█▄  ▁  ▄ ▄
      ▆▁▆▆▆▁▆▆█▆▆▁▆▁▁▆▆▁█▁▁▁▁▁▆▁█▆▁▁▁▆▁█▁▆▁▆▁▁▆▆██▆█▁▆███▆▆█▆▆█▁█ ▁
      73 ms           Histogram: frequency by time        92.1 ms <

     Memory estimate: 47.06 MiB, allocs estimate: 788896.

```julia

```


# Logistic Regression tutorial
The science is is data science:
> Given raw data, constraints, and a problem statement, how to determine a solution.

Main steps:
1. You hold on to your existing best performer.
2. Once you have a new idea to prototype, set up an experiment wherein the two best models compete.
3. Rinse and repeat (while not overfitting).

## Classifiers
Classification involves mapping your data points to a finite set of labels or the probability of a given label or labels.

In building classifiers, the problem is: *given data, a real-world classification problem, and constraints, you need to determine:
1. Which classifier to use
2. Which optimization method to employ
3. Which loss function to minimize
4. Which features to take from the data
5. Which evaluation metric to use

**Focus of this case study**: User-level conversion prediction - forecasting when someone will click using logistic regression

**Difference between logistic regression and linear regression**
The aim of logistic regression is o output a probability while that of linear regression is to predict the actual value.

### The Underlying Math
You want a function that takes the data and transforms it into a single value bounded inside the closed interval $[0,1]$. An example is the inverse-logit function:
\begin{equation}
    P(t)=logit^{-1}(t)\equiv\frac{1}{\left(1+e^{-t}\right)}=\frac{e^t}{1+e^t}
\end{equation}

$logit(P(c=1\mid x_i))=\alpha+\beta^\tau x_i$, where $c_i$ is the labels or classes, and $x_i$ is teh vector of features for user i.

The parameters $\alpha$ and $\beta$ could be estimated using Maximum Likelihood Estimation (MLE):
\begin{equation}
\Theta_{MLE} = \underset{\Theta}{argmax}\prod ^n_1p^{c_i}_i\cdot\left(1-p_i\right)^{1-c_i}
\end{equation}
#### Evaluation
Some ways of evaluating the model:
- **Area under the receiver operating curve (AUC):** ranks points, according to varied thresholds, into two categories of a binary classification problem (e.g., positive or negative)
- **Area under the cumulative lift curve:** measures how many times it is better to use a model versus not using the model (i.e., just selecting at random)

Others are: accuracy, precision, recall, F-score - a harmonic mean of precision and recall.

For density estimation: mean squared error, root squared error, mean aboslute error.

### Tutorial details

**problem:** Given the data of a marketing campaigns/phone calls by a Portuguese banking institution, predict whether a client will subscribe to a term deposit or not. The dataset provides the bank customers' information and a label column for customer subscription status.
- This codes from this tutorial is followed from [here](https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8).
- The dataset can be found [here.](https://raw.githubusercontent.com/madmashup/targeted-marketing-predictive-engine/master/banking.csv)

```python
import pandas as pd
import numpy as np
from sklearn import preprocessing
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import seaborn as sns
```

```python
plt.rc("font", size=14)
sns.set(style="white")
sns.set(style="whitegrid", color_codes=True)
%matplotlib inline
```

```python
data = pd.read_csv('banking.csv', header=0)
```

```python
data.dropna()
print(data.shape)
print(data.columns)
```

    (41188, 21)
    Index(['age', 'job', 'marital', 'education', 'default', 'housing', 'loan',
           'contact', 'month', 'day_of_week', 'duration', 'campaign', 'pdays',
           'previous', 'poutcome', 'emp_var_rate', 'cons_price_idx',
           'cons_conf_idx', 'euribor3m', 'nr_employed', 'y'],
          dtype='object')

```python
data.head()
```

      age
      job
      marital
      education
      default
      housing
      loan
      contact
      month
      day_of_week
      ...
      campaign
      pdays
      previous
      poutcome
      emp_var_rate
      cons_price_idx
      cons_conf_idx
      euribor3m
      nr_employed
      y

      0
      44
      blue-collar
      married
      basic.4y
      unknown
      yes
      no
      cellular
      aug
      thu
      ...
      1
      999
      0
      nonexistent
      1.4
      93.444
      -36.1
      4.963
      5228.1
      0

      1
      53
      technician
      married
      unknown
      no
      no
      no
      cellular
      nov
      fri
      ...
      1
      999
      0
      nonexistent
      -0.1
      93.200
      -42.0
      4.021
      5195.8
      0

      2
      28
      management
      single
      university.degree
      no
      yes
      no
      cellular
      jun
      thu
      ...
      3
      6
      2
      success
      -1.7
      94.055
      -39.8
      0.729
      4991.6
      1

      3
      39
      services
      married
      high.school
      no
      no
      no
      cellular
      apr
      fri
      ...
      2
      999
      0
      nonexistent
      -1.8
      93.075
      -47.1
      1.405
      5099.1
      0

      4
      55
      retired
      married
      basic.4y
      no
      yes
      no
      cellular
      aug
      fri
      ...
      1
      3
      1
      success
      -2.9
      92.201
      -31.4
      0.869
      5076.2
      1

5 rows × 21 columns

```python
data['education'].unique()
```

    array(['basic.4y', 'unknown', 'university.degree', 'high.school',
           'basic.9y', 'professional.course', 'basic.6y', 'illiterate'],
          dtype=object)

```python
data['education']=np.where(data['education']=='basic.9y','basic', data['education'])
data['education']=np.where(data['education']=='basic.4y','basic', data['education'])
data['education']=np.where(data['education']=='basic.6y','basic', data['education'])
```

```python
data['education'].unique()
```

    array(['basic', 'unknown', 'university.degree', 'high.school',
           'professional.course', 'illiterate'], dtype=object)

```python
data['y'].value_counts()
```

    0    36548
    1     4640
    Name: y, dtype: int64

```python
sns.countplot(x='y',data=data, palette='hls')
```

```python
count_no_sub = len(data[data['y']==0])
count_sub = len(data[data['y']==1])
pct_of_no_sub = count_no_sub/(count_no_sub+count_sub)
print("percentage of no subscription is", pct_of_no_sub*100)
pct_of_sub = count_sub/(count_no_sub+count_sub)
print("percentage of subscription", pct_of_sub*100)
```

    percentage of no subscription is 88.73458288821988
    percentage of subscription 11.265417111780131

```python
data.groupby('y').mean()
```

      age
      duration
      campaign
      pdays
      previous
      emp_var_rate
      cons_price_idx
      cons_conf_idx
      euribor3m
      nr_employed

      y

      0
      39.911185
      220.844807
      2.633085
      984.113878
      0.132374
      0.248875
      93.603757
      -40.593097
      3.811491
      5176.166600

      1
      40.913147
      553.191164
      2.051724
      792.035560
      0.492672
      -1.233448
      93.354386
      -39.789784
      2.123135
      5095.115991

```python
data.groupby('job').mean()
```

      age
      duration
      campaign
      pdays
      previous
      emp_var_rate
      cons_price_idx
      cons_conf_idx
      euribor3m
      nr_employed
      y

      job

      admin.
      38.187296
      254.312128
      2.623489
      954.319229
      0.189023
      0.015563
      93.534054
      -40.245433
      3.550274
      5164.125350
      0.129726

      blue-collar
      39.555760
      264.542360
      2.558461
      985.160363
      0.122542
      0.248995
      93.656656
      -41.375816
      3.771996
      5175.615150
      0.068943

      entrepreneur
      41.723214
      263.267857
      2.535714
      981.267170
      0.138736
      0.158723
      93.605372
      -41.283654
      3.791120
      5176.313530
      0.085165

      housemaid
      45.500000
      250.454717
      2.639623
      960.579245
      0.137736
      0.433396
      93.676576
      -39.495283
      4.009645
      5179.529623
      0.100000

      management
      42.362859
      257.058140
      2.476060
      962.647059
      0.185021
      -0.012688
      93.522755
      -40.489466
      3.611316
      5166.650513
      0.112175

      retired
      62.027326
      273.712209
      2.476744
      897.936047
      0.327326
      -0.698314
      93.430786
      -38.573081
      2.770066
      5122.262151
      0.252326

      self-employed
      39.949331
      264.142153
      2.660802
      976.621393
      0.143561
      0.094159
      93.559982
      -40.488107
      3.689376
      5170.674384
      0.104856

      services
      37.926430
      258.398085
      2.587805
      979.974049
      0.154951
      0.175359
      93.634659
      -41.290048
      3.699187
      5171.600126
      0.081381

      student
      25.894857
      283.683429
      2.104000
      840.217143
      0.524571
      -1.408000
      93.331613
      -40.187543
      1.884224
      5085.939086
      0.314286

      technician
      38.507638
      250.232241
      2.577339
      964.408127
      0.153789
      0.274566
      93.561471
      -39.927569
      3.820401
      5175.648391
      0.108260

      unemployed
      39.733728
      249.451677
      2.564103
      935.316568
      0.199211
      -0.111736
      93.563781
      -40.007594
      3.466583
      5157.156509
      0.142012

      unknown
      45.563636
      239.675758
      2.648485
      938.727273
      0.154545
      0.357879
      93.718942
      -38.797879
      3.949033
      5172.931818
      0.112121

```python
data.groupby('marital').mean()
```

      age
      duration
      campaign
      pdays
      previous
      emp_var_rate
      cons_price_idx
      cons_conf_idx
      euribor3m
      nr_employed
      y

      marital

      divorced
      44.899393
      253.790330
      2.61340
      968.639853
      0.168690
      0.163985
      93.606563
      -40.707069
      3.715603
      5170.878643
      0.103209

      married
      42.307165
      257.438623
      2.57281
      967.247673
      0.155608
      0.183625
      93.597367
      -40.270659
      3.745832
      5171.848772
      0.101573

      single
      33.158714
      261.524378
      2.53380
      949.909578
      0.211359
      -0.167989
      93.517300
      -40.918698
      3.317447
      5155.199265
      0.140041

      unknown
      40.275000
      312.725000
      3.18750
      937.100000
      0.275000
      -0.221250
      93.471250
      -40.820000
      3.313038
      5157.393750
      0.150000

```python
table = pd.crosstab(data.job, data.y).plot(kind='bar')
```

```python
table=pd.crosstab(data.marital,data.y)
table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Stacked Bar Chart of Marital Status vs Purchase')
plt.xlabel('Marital Status')
plt.ylabel('Proportion of Customers')
plt.savefig('mariral_vs_pur_stack')
```

```python
table=pd.crosstab(data.education,data.y)
table.div(table.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Stacked Bar Chart of Education vs Purchase')
plt.xlabel('Education')
plt.ylabel('Proportion of Customers')
plt.savefig('edu_vs_pur_stack')
```

```python
pd.crosstab(data.day_of_week,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Day of Week')
plt.xlabel('Day of Week')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_dayofweek_bar')
```

```python
pd.crosstab(data.month,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Month')
plt.xlabel('Month')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_fre_month_bar')
```

```python
data.age.hist()
plt.title('Histogram of Age')
plt.xlabel('Age')
plt.ylabel('Frequency')
plt.savefig('hist_age')
```

```python
pd.crosstab(data.poutcome,data.y).plot(kind='bar')
plt.title('Purchase Frequency for Poutcome')
plt.xlabel('Poutcome')
plt.ylabel('Frequency of Purchase')
plt.savefig('pur_fre_pout_bar')
```

###  Create dummy variables

```python
cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']
for var in cat_vars:
    cat_list='var'+'_'+var
    cat_list = pd.get_dummies(data[var], prefix=var)
    data1=data.join(cat_list)
    data=data1
cat_vars=['job','marital','education','default','housing','loan','contact','month','day_of_week','poutcome']
data_vars=data.columns.values.tolist()
to_keep=[i for i in data_vars if i not in cat_vars]
```

```python
data_final = data[to_keep]
```

```python
data_final.columns.values
```

    array(['age', 'duration', 'campaign', 'pdays', 'previous', 'emp_var_rate',
           'cons_price_idx', 'cons_conf_idx', 'euribor3m', 'nr_employed', 'y',
           'job_admin.', 'job_blue-collar', 'job_entrepreneur',
           'job_housemaid', 'job_management', 'job_retired',
           'job_self-employed', 'job_services', 'job_student',
           'job_technician', 'job_unemployed', 'job_unknown',
           'marital_divorced', 'marital_married', 'marital_single',
           'marital_unknown', 'education_basic', 'education_high.school',
           'education_illiterate', 'education_professional.course',
           'education_university.degree', 'education_unknown', 'default_no',
           'default_unknown', 'default_yes', 'housing_no', 'housing_unknown',
           'housing_yes', 'loan_no', 'loan_unknown', 'loan_yes',
           'contact_cellular', 'contact_telephone', 'month_apr', 'month_aug',
           'month_dec', 'month_jul', 'month_jun', 'month_mar', 'month_may',
           'month_nov', 'month_oct', 'month_sep', 'day_of_week_fri',
           'day_of_week_mon', 'day_of_week_thu', 'day_of_week_tue',
           'day_of_week_wed', 'poutcome_failure', 'poutcome_nonexistent',
           'poutcome_success'], dtype=object)

###  Solving class imbalance using SMOTE (Synthetic Minority Oversampling Technique)

```python
X=data_final.loc[:, data_final.columns != 'y']
y=data_final.loc[:, data_final.columns == 'y']
```

```python
from imblearn.over_sampling import SMOTE
os = SMOTE(random_state=0)
```

```python
X_train, X_test, y_train, y_test=train_test_split(X, y, test_size=0.3, random_state=0)
columns = X_train.columns
```

```python
os_data_X,os_data_y=os.fit_sample(X_train, y_train)
os_data_X = pd.DataFrame(data=os_data_X,columns=columns )
os_data_y= pd.DataFrame(data=os_data_y,columns=['y'])
```

    /home/bbrighttaer/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:761: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)

```python
os_data_X.head()
```

      age
      duration
      campaign
      pdays
      previous
      emp_var_rate
      cons_price_idx
      cons_conf_idx
      euribor3m
      nr_employed
      ...
      month_oct
      month_sep
      day_of_week_fri
      day_of_week_mon
      day_of_week_thu
      day_of_week_tue
      day_of_week_wed
      poutcome_failure
      poutcome_nonexistent
      poutcome_success

      0
      31.0
      259.0
      1.0
      999.0
      1.0
      -0.1
      93.200
      -42.0
      4.153
      5195.8
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0

      1
      44.0
      300.0
      1.0
      999.0
      0.0
      1.1
      93.994
      -36.4
      4.857
      5191.0
      ...
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0

      2
      43.0
      134.0
      1.0
      999.0
      0.0
      1.1
      93.994
      -36.4
      4.857
      5191.0
      ...
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0

      3
      35.0
      169.0
      2.0
      999.0
      0.0
      -0.1
      93.200
      -42.0
      4.153
      5195.8
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0

      4
      33.0
      255.0
      2.0
      999.0
      1.0
      -1.8
      92.893
      -46.2
      1.266
      5099.1
      ...
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0

5 rows × 61 columns

```python
# we can Check the numbers of our data
print("length of oversampled data is ",len(os_data_X))
print("Number of no subscription in oversampled data",len(os_data_y[os_data_y['y']==0]))
print("Number of subscription",len(os_data_y[os_data_y['y']==1]))
print("Proportion of no subscription data in oversampled data is ",len(os_data_y[os_data_y['y']==0])/len(os_data_X))
print("Proportion of subscription data in oversampled data is ",len(os_data_y[os_data_y['y']==1])/len(os_data_X))
```

    length of oversampled data is  51134
    Number of no subscription in oversampled data 25567
    Number of subscription 25567
    Proportion of no subscription data in oversampled data is  0.5
    Proportion of subscription data in oversampled data is  0.5

### Recursive Feature Elimination

```python
data_final_vars = data_final.columns.values.tolist()
y=['y']
X=[i for i in data_final_vars if i not in y]
```

```python
len(X)
```

    61

```python
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
```

```python
logreg = LogisticRegression(solver='liblinear')
```

```python
rfe = RFE(logreg, 20)
```

```python
rfe = rfe.fit(os_data_X, os_data_y.values.ravel())
print(rfe.support_)
print(rfe.ranking_)
```

    [False False False False False False False False  True False False  True
     False  True False False  True False False False False False False False
     False  True False False  True False False False  True  True False False
     False False False False False  True  True  True  True  True  True  True
     False  True  True False  True False False False False False  True False
      True]
    [39 36 24 40  9 13 22 35  1 31  7  1  6  1  4 30  1  3 27  2  5 10 20 19
     15  1 14 18  1 17 16 21  1  1 41 25 38 33 29 37 32  1  1  1  1  1  1  1
     12  1  1 34  1 28 11 26 42 23  1  8  1]

```python
cols = [os_data_X.columns.values[i] for i,v in enumerate(rfe.support_.tolist()) if bool(v) is True]
```

```python
X=os_data_X[cols]
y=os_data_y['y']
```

###  Implementing the model

```python
import statsmodels.api as sm
```

```python
logit_model = sm.Logit(y, X)
```

```python
result = logit_model.fit()
```

    Warning: Maximum number of iterations has been exceeded.
             Current function value: 0.545860
             Iterations: 35

    /home/bbrighttaer/anaconda3/lib/python3.6/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals
      "Check mle_retvals", ConvergenceWarning)

```python
print(result.summary2())
```

                                    Results: Logit
    ==============================================================================
    Model:                   Logit                Pseudo R-squared:     0.212
    Dependent Variable:      y                    AIC:                  55863.9960
    Date:                    2019-01-10 00:27     BIC:                  56040.8401
    No. Observations:        51134                Log-Likelihood:       -27912.
    Df Model:                19                   LL-Null:              -35443.
    Df Residuals:            51114                LLR p-value:          0.0000
    Converged:               0.0000               Scale:                1.0000
    No. Iterations:          35.0000
    ------------------------------------------------------------------------------
                          Coef.    Std.Err.    z     P>|z|     [0.025     0.975]
    ------------------------------------------------------------------------------
    euribor3m             -0.4637    0.0091 -50.9870 0.0000     -0.4815    -0.4459
    job_blue-collar       -0.1841    0.0285  -6.4615 0.0000     -0.2399    -0.1283
    job_housemaid         -0.3352    0.0779  -4.3054 0.0000     -0.4878    -0.1826
    job_self-employed     -0.1539    0.0613  -2.5127 0.0120     -0.2740    -0.0339
    marital_unknown        0.7487    0.2253   3.3232 0.0009      0.3071     1.1902
    education_illiterate   1.3600    0.4389   3.0990 0.0019      0.4999     2.2202
    default_no            16.1545 5406.2084   0.0030 0.9976 -10579.8193 10612.1282
    default_unknown       15.8968 5406.2084   0.0029 0.9977 -10580.0770 10611.8706
    contact_cellular     -13.3581 5406.2084  -0.0025 0.9980 -10609.3319 10582.6157
    contact_telephone    -13.4276 5406.2084  -0.0025 0.9980 -10609.4013 10582.5462
    month_apr             -1.4097    0.0696 -20.2461 0.0000     -1.5462    -1.2732
    month_aug             -1.2663    0.0694 -18.2586 0.0000     -1.4022    -1.1304
    month_dec             -0.9963    0.1531  -6.5072 0.0000     -1.2964    -0.6962
    month_jul             -0.9787    0.0695 -14.0733 0.0000     -1.1150    -0.8424
    month_jun             -1.0535    0.0673 -15.6494 0.0000     -1.1854    -0.9215
    month_may             -2.0478    0.0627 -32.6745 0.0000     -2.1706    -1.9249
    month_nov             -1.4031    0.0711 -19.7237 0.0000     -1.5425    -1.2637
    month_sep             -0.5806    0.1041  -5.5761 0.0000     -0.7847    -0.3765
    poutcome_failure      -0.5039    0.0363 -13.8884 0.0000     -0.5751    -0.4328
    poutcome_success       1.5751    0.0618  25.4749 0.0000      1.4539     1.6963
    ==============================================================================

The p-values for most of the variables are smaller than 0.05, except four variables, therefore, we will remove them.

```python
cols=['euribor3m', 'job_blue-collar', 'job_housemaid', 'marital_unknown', 'education_illiterate',
      'month_apr', 'month_aug', 'month_dec', 'month_jul', 'month_jun', 'month_mar',
      'month_may', 'month_nov', 'month_oct', "poutcome_failure", "poutcome_success"]
X=os_data_X[cols]
y=os_data_y['y']
```

```python
logit_model=sm.Logit(y,X)
result=logit_model.fit()
print(result.summary2())
```

    Optimization terminated successfully.
             Current function value: 0.555890
             Iterations 7
                               Results: Logit
    =====================================================================
    Model:                Logit             Pseudo R-squared:  0.198
    Dependent Variable:   y                 AIC:               56881.7793
    Date:                 2019-01-10 00:29  BIC:               57023.2546
    No. Observations:     51134             Log-Likelihood:    -28425.
    Df Model:             15                LL-Null:           -35443.
    Df Residuals:         51118             LLR p-value:       0.0000
    Converged:            1.0000            Scale:             1.0000
    No. Iterations:       7.0000
    ---------------------------------------------------------------------
                          Coef.  Std.Err.    z     P>|z|   [0.025  0.975]
    ---------------------------------------------------------------------
    euribor3m            -0.4488   0.0074 -60.6838 0.0000 -0.4632 -0.4343
    job_blue-collar      -0.2097   0.0278  -7.5345 0.0000 -0.2643 -0.1552
    job_housemaid        -0.2793   0.0762  -3.6636 0.0002 -0.4287 -0.1299
    marital_unknown       0.7618   0.2244   3.3953 0.0007  0.3220  1.2016
    education_illiterate  1.3085   0.4346   3.0108 0.0026  0.4567  2.1604
    month_apr             1.2863   0.0380  33.8252 0.0000  1.2117  1.3608
    month_aug             1.3948   0.0411  33.9435 0.0000  1.3143  1.4754
    month_dec             1.8090   0.1441  12.5523 0.0000  1.5265  2.0915
    month_jul             1.6763   0.0424  39.5459 0.0000  1.5932  1.7594
    month_jun             1.5586   0.0408  38.1651 0.0000  1.4786  1.6387
    month_mar             2.8220   0.0908  31.0943 0.0000  2.6442  2.9999
    month_may             0.5867   0.0304  19.2814 0.0000  0.5270  0.6463
    month_nov             1.2735   0.0445  28.5964 0.0000  1.1863  1.3608
    month_oct             2.7225   0.0815  33.3924 0.0000  2.5627  2.8823
    poutcome_failure     -0.2803   0.0351  -7.9904 0.0000 -0.3490 -0.2115
    poutcome_success      1.9619   0.0602  32.5957 0.0000  1.8439  2.0798
    =====================================================================

```python
from sklearn import metrics
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
```

    /home/bbrighttaer/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
      FutureWarning)

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='warn',
              n_jobs=None, penalty='l2', random_state=None, solver='warn',
              tol=0.0001, verbose=0, warm_start=False)

```python
y_pred = logreg.predict(X_test)
```

```python
y_pred
```

    array([0, 1, 1, ..., 0, 1, 0])

```python
print('Accuracy of logistic regression classifier on test set: {:.2f}'.format(logreg.score(X_test, y_test)))
```

    Accuracy of logistic regression classifier on test set: 0.74

```python
from sklearn.metrics import confusion_matrix
confusion_matrix = confusion_matrix(y_test, y_pred)
print(confusion_matrix)
```

    [[6124 1542]
     [2505 5170]]

```python
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
```

                  precision    recall  f1-score   support

               0       0.71      0.80      0.75      7666
               1       0.77      0.67      0.72      7675

       micro avg       0.74      0.74      0.74     15341
       macro avg       0.74      0.74      0.74     15341
    weighted avg       0.74      0.74      0.74     15341

```python
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_curve
logit_roc_auc = roc_auc_score(y_test, logreg.predict(X_test))
fpr, tpr, thresholds = roc_curve(y_test, logreg.predict_proba(X_test)[:,1])
plt.figure()
plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
# plt.savefig('Log_ROC')
plt.show()
```

```python

```