# Clase 3b: Perfil de Yukovski

_Aunque no te lo creas, __con lo que hemos visto hasta ahora eres capaz de hacer grandes cosas__. Vale sí, un perfil de Yukovski no es gran cosa aerodinámicamente, pero si lo hacemos en Python... Echa un vistazo a la figura ¿no está mal, no? algo así intentaremos conseguir al final de esta clase._

_Como no se trata de aprender (o reaprender) aerodinámica, te daremos las funciones matemáticas y los pasos a seguir así como la estructura del programa. Tú sólo tienes que preocuparte de programar cada bloque. Puedes leer en detalle todo lo relativo a la aerodinámica en el libro  Aerodinámica básica de Meseguer Ruiz, J., Sanz Andrés, A. (Editorial Garceta)._

## 1. Importamos paquetes.

Lo primero es lo primero, importemos los paquetes:

```python
# Recuerda, utilizaremos arrays y pintaremos gráficas.
import numpy as np
import matplotlib.pyplot as plt
```

## 2. Parámetros del problema

######
__Fuente:__ _Aerodinámica básica, Meseguer Ruiz, J., Sanz Andrés, A._

La transformación de Yukovski es: $$\tau=t+\frac{a^{2}}{t}$$

Los parámetros del problema son los del siguiente bloque, puedes cambiarlos más adelante:

```python
# Datos para el perfil de Yukovski
# Parámetro de la transformación de Yukovski
a = 1

# Centro de la circunferencia
landa = 0.2    # coordenada x (en valor absoluto)
delta = 0.3     # coordenada y

t0 = a * (-landa + delta * 1j)     # centro: plano complejo

# Valor del radio de la circunferencia
R = a * np.sqrt((1 + landa)**2 + delta**2)

# Ángulo de ataque corriente incidente
alfa_grados = 0
alfa = np.deg2rad(alfa_grados)

#Velocidad de la corriente incidente
U = 1
```

## 3. Perfil de Yukoski a partir de una circunferencia.

### Función transformación de Yukovski

__Se trata de definir una función que realice la transformación de Yukovski.__ Esta función recibirá el parámetro de la transformación, $a$ y el punto del plano complejo $t$. Devolverá el valor $\tau$, punto del plano complejo en el que se transforma $t$.

```python
def transf_yukovski(a, t):
    """Dado el punto t (complejo) y el parámetro a
    a de la transformación proporciona el punto
    tau (complejo) en el que se transforma t."""
    tau = t + a ** 2 / t
    return tau
```

```python
# aeropython: preserve
#comprobamos que la función está bien programada
#puntos del eje real siguen siendo del eje real
err_message = "La transformación de Yukovski no devuelve un resultado correcto"
np.testing.assert_equal(transf_yukovski(1, 1+0j), 2+0j, err_message)
```

### Circunferencia

Ahora queremos transformar la circunferencia de radio $R$ con centro en $t_0$ usando la función anterior:

1. __Creamos `N` puntos de la circunferencia__ de modo que __en `Xc` estén las coordenadas $x$ y en `Yc` estén las coordenadas $y$__ de los puntos que la forman. Controla el número de puntos mediante un parámetro que se llame `N_perfil`.
    $$X_c = real(t_0) + R·cos(\theta)$$
    $$Y_c = imag(t_0) + R·sin(\theta)$$
2. Una vez hayas obtenido los dos arrays `Xc` e `Yc`, __píntalos mediante un `scatter`__ para comprobar que todo ha ido bien.
3. Pinta también el __centro de la circunferencia__.

Deberías obtener algo así:

```python

# Número de puntos de la circunferencia que
# vamos a transformar para obtener el perfil
N_perfil = 100

#se barre un ángulo de 0 a 2 pi
theta = np.linspace(0, 2*np.pi, N_perfil)

#se crean las coordenadas del los puntos
#de la circunferencia
Xc = np.real(t0) + R * np.cos(theta)
Yc = np.imag(t0) + R * np.sin(theta)

#lo visualizamos
plt.figure("circunferencia", figsize=(5,5))
plt.title('Circunferencia', {'fontsize':20})
plt.scatter(Xc, Yc)
plt.scatter(np.real(t0), np.imag(t0), color='orange', marker='x', s=100)
plt.grid()
plt.show()
```

```python
# aeropython: preserve
# Lo visualizamos más bonito
plt.figure("circunferencia", figsize=(5,5))
plt.title('Circunferencia', {'fontsize':20})
# Esto no tienes por qué entenderlo ahora
p = plt.Polygon(list(zip(Xc, Yc)), color="#cccccc", zorder=3)
plt.gca().add_patch(p)

plt.ylim(-1.5, 2)
plt.xlim(-2, 1.5)
plt.grid()
plt.show()
```

### Transformación de cirunferencia a perfil

Ahora estamos en condiciones de __transformar estos puntos de la circunferencia (`Xc`, `Yc`) en los del perfil (`Xp`, `Yp`)__. Para esto vamos a usar nuestra función `transf_yukovski`. Recuerda que esta función recibe y da números complejos. ¿Saldrá un perfil?

```python
Puntos_perfil = transf_yukovski(a, Xc+Yc*1j)
Xp, Yp = np.real(Puntos_perfil) , np.imag(Puntos_perfil)

#lo visualizamos
plt.figure("perfil yukovski", figsize=(10,10))
plt.title('Perfil', {'fontsize':20})
plt.scatter(Xp, Yp)
plt.grid()
plt.gca().set_aspect(1)
plt.show()
```

```python
# aeropython: preserve
#lo visualizamos más bonito
plt.figure('perfil yukovski', figsize=(10,10))
plt.title('Perfil', {'fontsize':20})

p = plt.Polygon(list(zip(Xp, Yp)), color="#cccccc", zorder=3)
plt.gca().add_patch(p)
plt.gca().set_aspect(1)

plt.xlim(-3, 3)
plt.ylim(-0.4,1)
plt.grid()

plt.show()
```

## 4. Flujo alrededor del cilindro

Para visualizar ahora el flujo alrededor del cilindro recurrimos al __potencial complejo__ de una _corriente uniforme_ que forme un ángulo $\alpha$ con el eje $x$ _en presencia de un cilindro_ (aplicando el teorema del círculo) y se añade un torbellino con la intensidad adecuada para que se cumpla la hipótesis de Kutta en el perfil:

\begin{equation}
f(t)=U_{\infty}\text{·}\left((t-t_{0})\text{·}e^{-i\alpha}+\frac{R^{2}}{t-t_{0}}\text{·}e^{i\alpha}\right)+\frac{i\Gamma}{2\pi}\text{·}ln(t-t_{0})=\Phi+i\Psi
\end{equation}

donde $\Phi$ es el potencial de velocidades y $\Psi$ es la función de corriente.

$$\Gamma = 4  \pi  a  U  (\delta + (1+\lambda)  \alpha)$$

$\Gamma$ es la circulación que hay que añadir al cilindro para que al transformarlo en el perfil se cumpla la condición de Kutta.

Recordando que la función de corriente toma un valor constante en las líneas de corriente, sabemos que: dibujando $\Psi=cte$ se puede visualizar el flujo.

__Pintaremos estas lineas de potencial constante utilizando la función `contour()`, pero antes tendremos que crear una malla circular. Esto será lo primero que hagamos:__

1. Crea un parámetro `N_R` cuyo valor sea el número de puntos que va a tener la malla en dirección radial. Desde otro punto de vista, esta parámetro es el número de círculos concéntricos que forman la malla.
2. Crea dos parámetros `R_min` y `R_max` que representen el radio mínimo y máximo entre los que se extiende la malla. El radio mínimo debe de ser el radio del círculo, porque estamos calculando el aire en el exterior del perfil.
4. La dirección tangencial necesita un sólo parámetro `N_T`, que representa el número de puntos que la malla tendrá en esta dirección. Dicho de otro modo, cuántos puntos forman los círculos concéntricos de la malla.
3. Crea un array `R_` que vaya desde `R_min` hasta `R_max` y que tenga `N_R` elementos. De manera análoga, crea el array `T_`, que al representar los ángulos de los puntos que forman las circunferencias, debe ir de 0 a 2$\pi$, y tener `N_T` elementos.
4. Para trabajar con la malla, deberemos usar coordenadas cartesianas. Crea la malla: `XX, YY` van a ser dos matrices de `N_T · N_R` elementos. Cada elemento de estas matrices se corresponde con un punto de la malla: la matriz `XX` contiene las coordenadas X de cada punto y la matriz `YY`, las coordenadas y.

La manera de generar estas matrices tiene un poco de truco, porque depende de ambos vectores.
Para cada elemento, $x = real(t_0) +  R · cos (T) $ ,  $y = imag(t_0) + R · sin(T)$.

__Recuerda que:__

* Los puntos sobre la circunferencia se transforman en el perfil.
* Los puntos interiores a la circunferencia se transforman en puntos interiores al perfil.
* Los puntos exteriores a la circunferencia se transforman en puntos exteriores al perfil. __Los puntos que nos interesan__.

```python
#se crea la malla donde se va pintar la función de corriente
# Dirección radial
N_R = 50   # Número de puntos en la dirección radial
R_min = R
R_max = 10

# Dirección tangencial
N_T = 180   # Número de puntos en la dirección tangencial

R_ = np.linspace(R_min, R_max, N_R)

T_ = np.linspace(0, 2*np.pi , N_T)

# Crear la malla:
XX =  R_ * np.cos(T_).reshape((-1, 1)) + np.real(t0)
YY = R_ * np.sin(T_).reshape((-1, 1)) + np.imag(t0)
```

```python
#pintamos la malla para verla
plt.figure(figsize=(10,10))
plt.scatter(XX.flatten(), YY.flatten(), marker='.')
plt.show()
```

__NOTA__: En versiones anteriores se utilizaba una malla rectangular. Esto generaba algunos problemas con los puntos interiores a la hora de pintar las líneas de corriente y los campos de velocidades y presiones. La idea de usar una malla circular está tomada de [este ejercicio](http://nbviewer.ipython.org/github/barbagroup/AeroPython/blob/master/lessons/06_Lesson06_Assignment.ipynb) del curso Aerodynamics-Hydrodynamics with Python de la [Prof. Lorena Barba](http://lorenabarba.com/).

### Probando a transformar la malla

Bueno, lo que queríamos era hacer cosas alrededor de nuestro perfil, ¿no?

Esto lo conseguiremos pintando la función $\Psi$ en los puntos `XX_tau, YY_tau` transformados de los `XX, YY` a través de la función `transf_yukovski`, recuerda que la transformación que tenemos recibe y da números complejos. Como antes, debes separar parte real e imaginaria. En la siguiente celda calcula y transforma `tt` (donde debe estar almacenada la malla en forma compleja) para obtener `XX_tau, YY_tau`.

Probemos a visualizar como se transforman los puntos de la malla primero.

```python
tt = XX + YY * 1j
tautau = transf_yukovski(a, tt)
XX_tau, YY_tau = np.real(tautau) , np.imag(tautau)
```

```python
# Comprobamos que los puntos exteriores a la circunferencia se transforman en los puntos exteriores del perfil
#pintamos la malla para verla
plt.figure(figsize=(10,10))
plt.scatter(XX_tau.flatten(), YY_tau.flatten(), marker='.')
plt.show()
```

### Obteniendo el flujo

1. Crea una variable `T` que tenga el valor correspondiente a la circulación $\Gamma$.
2. Utilizando el array `tt`, el valor `T` y los parámetros definidos al principio (`t0, alfa, U...`) crea `f` según la fórmula de arriba (no hace falta que crees una función).
3. Guarda la parte imaginaria de esa función (función de corriente) en una variable `psi`.

```python
# aeropython: preserve
# Circulación que hay que añadir al cilindro para
# que se cumpla la hipótesis de Kutta en el perfil
T = 4 * np.pi * a * U * (delta + (1+landa) * alfa)

# Malla compleja
tt = XX + YY * 1j

# Potencial complejo
f = U * ( (tt - t0) * np.exp(-alfa *1j) + R**2 / (tt - t0) * np.exp(alfa * 1j) )
f += 1j * T / (2* np.pi) * np.log(tt - t0)

# Función de corriente
psi = np.imag(f)
```

Como la función de corriente toma un valor constante en cada línea de corriente, podemos visualizar el flujo alrededor del cilindro pintando las lineas en las que `psi` toma un valor constante. Para ello utilizaremos la función `contour()` en la malla `XX, YY`. Si no se ve nada prueba a cambiar el número de líneas y los valores máximo y mínimo de la función que se representan.

```python
#lo visualizamos
# aeropython: preserve
plt.figure('lineas de corriente', figsize=(10,10))

plt.contour(XX, YY, psi, np.linspace(-5,5,50))
plt.grid()
plt.gca().set_aspect(1)
#plt.xlim(-8, 8)
#plt.ylim(-3, 3)

plt.show()
```

```python
#ponemos el cilindro encima
# aeropython: preserve
plt.figure('flujo cilindro', figsize=(10,10))

plt.contour(XX, YY, psi, np.linspace(-5,5,50), colors=['blue', 'blue'])
plt.grid()
plt.gca().set_aspect(1)

p = plt.Polygon(list(zip(Xc, Yc)), color="#cccccc", zorder=3)
plt.gca().add_patch(p)
plt.show()
```

## 5. Flujo alrededor del perfil

```python
plt.figure("flujo perfil", figsize=(12,12))

plt.contour(XX_tau, YY_tau, psi, np.linspace(-5,5,50))

plt.xlim(-8,8)
plt.ylim(-3,3)
plt.grid()
plt.gca().set_aspect(1)

plt.show()
```

```python
#Ahora ponemos el perfil encima
# aeropython: preserve
plt.figure("flujo perfil", figsize=(12,12))

plt.contour(XX_tau, YY_tau, psi, np.linspace(-5, 5, 50), colors=['blue', 'blue'])

p = plt.Polygon(list(zip(Xp, Yp)), color="#cccccc", zorder=10)
plt.gca().add_patch(p)

plt.xlim(-8,8)
plt.ylim(-3,3)
plt.grid()
plt.gca().set_aspect(1)
plt.show()
```

## 6. Interact

__Ahora es un buen momento para jugar con todos los parámetros del problema.
~~¡Prueba a cambiarlos y ejecuta el notebook entero!~~__

__Vamos a usar un `interact`, ¿no?__

Tenemos que crear una función que haga todas las tareas: reciba los argumentos y pinte para llamar a interact con ella. No tenemos más que cortar y pegar.

```python
# aeropython: preserve
def transformacion_geometrica(a, landa, delta, N_perfil=100):
    #punto del plano complejo
    t0 = a * (-landa + delta * 1j)
    #valor del radio de la circunferencia
    R = a * np.sqrt((1 + landa)**2 + delta**2)
    #se barre un ángulo de 0 a 2 pi
    theta = np.linspace(0, 2*np.pi, N_perfil)
    #se crean las coordenadas del los puntos
    #de la circunferencia
    Xc = - a * landa + R * np.cos(theta)
    Yc =   a * delta + R * np.sin(theta)
    #se crean las coordenadas del los puntos
    #del perfil
    Puntos_perfil = transf_yukovski(a, Xc+Yc*1j)
    Xp, Yp = np.real(Puntos_perfil) , np.imag(Puntos_perfil)

    #Se pintan la cirunferencia y el perfil
    fig, ax = plt.subplots(1,2)
    fig.set_size_inches(15,15)
    p_c = plt.Polygon(list(zip(Xc, Yc)), color="#cccccc", zorder=1)
    ax[0].add_patch(p_c)
    ax[0].plot(Xc,Yc)
    ax[0].set_aspect(1)
    ax[0].set_xlim(-3, 3)
    ax[0].set_ylim(-2,2)
    ax[0].grid()

    p_p = plt.Polygon(list(zip(Xp, Yp)), color="#cccccc", zorder=1)
    ax[1].add_patch(p_p)
    ax[1].plot(Xp,Yp)
    ax[1].set_aspect(1)

    ax[1].set_xlim(-3, 3)
    ax[1].set_ylim(-2,2)
    ax[1].grid()
    plt.show()
```

```python
from ipywidgets import interact
```

```python
# aeropython: preserve
interact(transformacion_geometrica, landa=(-1.,1, 0.01), delta=(-1.,1,0.01),
             a=(0,2.,0.1), N_perfil=(4, 200) )
```

    interactive(children=(FloatSlider(value=1.0, description='a', max=2.0), FloatSlider(value=0.0, description='la…

    <function __main__.transformacion_geometrica(a, landa, delta, N_perfil=100)>

```python
# aeropython: preserve
def flujo_perfil_circunferencia(landa, delta, alfa, U=1,  N_malla = 100):
    N_perfil=N_malla
    a=1
    #punto del plano complejo
    t0 = a * (-landa + delta * 1j)
    #valor del radio de la circunferencia
    R = a * np.sqrt((1 + landa)**2 + delta**2)

    #se barre un ángulo de 0 a 2 pi
    theta = np.linspace(0, 2*np.pi, N_perfil)
    #se crean las coordenadas del los puntos
    #de la circunferencia
    Xc = - a * landa + R * np.cos(theta)
    Yc =   a * delta + R * np.sin(theta)

    #se crean las coordenadas del los puntos
    #del perfil
    Puntos_perfil = transf_yukovski(a, Xc+Yc*1j)
    Xp, Yp = np.real(Puntos_perfil) , np.imag(Puntos_perfil)

    #se crea la malla donde se va pintar la función de corriente

    # Dirección radial
    N_R = N_malla//2   # Número de puntos en la dirección radial
    R_min = R
    R_max = 10

    # Dirección tangencial
    N_T = N_malla   # Número de puntos en la dirección tangencial

    R_ = np.linspace(R_min, R_max, N_R)
    T_ = np.linspace(0, 2*np.pi , N_T)

    # El menos en la XX  es para que el borde de ataque del perfil esté en la izquierda
    XX = - (R_ * np.cos(T_).reshape((-1, 1)) - np.real(t0))
    YY = R_ * np.sin(T_).reshape((-1, 1)) + np.imag(t0)

    tt = XX + YY * 1j

    alfa = np.deg2rad(alfa)
    # Circulación que hay que añadir al cilindro para
    # que se cumpla la hipótesis de Kutta en el perfil
    T = 4 * np.pi * a * U * (delta + (1+landa) * alfa)
    #Potencial complejo
    f = U * ( (tt - t0) * np.exp(-alfa *1j) + R**2 / (tt - t0) * np.exp(alfa * 1j) )
    f += 1j * T / (2* np.pi) * np.log(tt - t0)
    #Función de corriente
    psi = np.imag(f)

    Puntos_plano_tau = transf_yukovski(a, tt)
    XX_tau, YY_tau = np.real(Puntos_plano_tau) , np.imag(Puntos_plano_tau)

    #Se pinta
    fig, ax = plt.subplots(1,2)
    #lineas de corriente
    fig.set_size_inches(15,15)

    ax[0].contour(XX, YY, psi, np.linspace(-10,10,50), colors = ['blue', 'blue'])
    ax[0].grid()
    ax[0].set_aspect(1)
    p = plt.Polygon(list(zip(Xc, Yc)), color="#cccccc", zorder=10)
    ax[0].add_patch(p)
    ax[0].set_xlim(-5, 5)
    ax[0].set_ylim(-2,2)

    ax[1].contour(XX_tau, YY_tau, psi, np.linspace(-10,10,50), colors = ['blue', 'blue'])
    ax[1].grid()
    ax[1].set_aspect(1)
    p = plt.Polygon(list(zip(Xp, Yp)), color="#cccccc", zorder=10)
    ax[1].add_patch(p)
    ax[1].set_xlim(-5, 5)
    ax[1].set_ylim(-2,2)
    plt.show()
```

```python
p = interact(flujo_perfil_circunferencia, landa=(0.,1), delta=(0.,1), alfa=(0, 30), U=(0,10), N_malla = (10,150))
```

    interactive(children=(FloatSlider(value=0.5, description='landa', max=1.0), FloatSlider(value=0.5, description…

## 7. Pintemos un poco más

Con los datos que ya hemos manejado, sin mucho más esfuerzo, podemos fácilmente pintar la velocidad y la presión del aire alrededor del perfil.

```python
# aeropython: preserve
#Velocidad conjugada
dfdt = U * ( 1 * np.exp(-alfa * 1j) - R**2 / (tt - t0)**2 * np.exp(alfa * 1j) )
dfdt += 1j * T / (2*np.pi) * 1 / (tt - t0)
#coeficiente de presion
cp = 1 - np.abs(dfdt)**2 / U**2
```

```python
# aeropython: preserve
cmap = plt.cm.RdBu
```

```python
# aeropython: preserve
#Se pinta
fig, ax = plt.subplots(1,3)
#lineas de corriente
fig.set_size_inches(15,15)
ax[0].contour(XX, YY, psi, np.linspace(-10,10,50), colors = ['blue', 'blue'])
ax[0].grid()
ax[0].set_aspect(1)
p = plt.Polygon(list(zip(Xc, Yc)), color="#cccccc", zorder=10)
ax[0].add_patch(p)
#Campo de velocidades
ax[1].contourf(XX, YY, np.abs(dfdt), 200, cmap=cmap)
p = plt.Polygon(list(zip(Xc, Yc)), color="#cccccc", zorder=10)
ax[1].set_title('campo de velocidades')
ax[1].add_patch(p)
ax[1].set_aspect(1)
ax[1].grid()
#campo de presiones
ax[2].contourf(XX, YY, cp, 200, cmap=cmap)
p = plt.Polygon(list(zip(Xc, Yc)), color="#cccccc", zorder=10)
ax[2].set_title('coeficiente de presión')
ax[2].add_patch(p)
ax[2].set_aspect(1)
ax[2].grid()
plt.show()
```

```python
#Se pinta
# aeropython: preserve
fig, ax = plt.subplots(1,3)
#lineas de corriente
fig.set_size_inches(15,15)
ax[0].contour(XX_tau, YY_tau, psi, np.linspace(-10,10,50), colors = ['blue', 'blue'])
ax[0].grid()
ax[0].set_aspect(1)
p = plt.Polygon(list(zip(Xp, Yp)), color="#cccccc", zorder=10)
ax[0].add_patch(p)
#Campo de velocidades
ax[1].contourf(XX_tau, YY_tau, np.abs(dfdt), 200, cmap=cmap)
p = plt.Polygon(list(zip(Xp, Yp)), color="#cccccc", zorder=10)
ax[1].set_title('campo de velocidades')
ax[1].add_patch(p)
ax[1].set_aspect(1)
ax[1].grid()
#campo de presiones
ax[2].contourf(XX_tau, YY_tau, cp, 200, cmap=cmap)
p = plt.Polygon(list(zip(Xp, Yp)), color="#cccccc", zorder=10)
ax[2].set_title('coeficiente de presión')
ax[2].add_patch(p)
ax[2].set_aspect(1)
ax[2].grid()
plt.show()
```

```python
# aeropython: preserve
def cp_perfil_circunferencia(landa, delta, alfa, U=1,  N_malla = 100):
    N_perfil=N_malla
    a=1
    #punto del plano complejo
    t0 = a * (-landa + delta * 1j)
    #valor del radio de la circunferencia
    R = a * np.sqrt((1 + landa)**2 + delta**2)

    #se barre un ángulo de 0 a 2 pi
    theta = np.linspace(0, 2*np.pi, N_perfil)
    #se crean las coordenadas del los puntos
    #de la circunferencia
    Xc = - a * landa + R * np.cos(theta)
    Yc =   a * delta + R * np.sin(theta)

    #se crean las coordenadas del los puntos
    #del perfil
    Puntos_perfil = transf_yukovski(a, Xc+Yc*1j)
    Xp, Yp = np.real(Puntos_perfil) , np.imag(Puntos_perfil)

    #se crea la malla donde se va pintar la función de corriente

    # Dirección radial
    N_R = N_malla//2   # Número de puntos en la dirección radial
    R_min = R
    R_max = 10

    # Dirección tangencial
    N_T = N_malla   # Número de puntos en la dirección tangencial

    R_ = np.linspace(R_min, R_max, N_R)
    T_ = np.linspace(0, 2*np.pi, N_T)

    # El menos en la XX  es para que el borde de ataque del perfil esté en la izquierda
    XX = - (R_ * np.cos(T_).reshape((-1, 1)) - np.real(t0))
    YY = R_ * np.sin(T_).reshape((-1, 1)) + np.imag(t0)

    tt = XX + YY * 1j

    alfa = np.deg2rad(alfa)
    # Circulación que hay que añadir al cilindro para
    # que se cumpla la hipótesis de Kutta en el perfil
    T = 4 * np.pi * a * U * (delta + (1+landa) * alfa)

    #Velocidad conjugada
    dfdt = U * ( 1 * np.exp(-alfa * 1j) - R**2 / (tt - t0)**2 * np.exp(alfa * 1j) )
    dfdt = dfdt + 1j * T / (2*np.pi) * 1 / (tt - t0)
    #coeficiente de presion
    cp = 1 - np.abs(dfdt)**2 / U**2

    Puntos_plano_tau = transf_yukovski(a, tt)
    XX_tau, YY_tau = np.real(Puntos_plano_tau) , np.imag(Puntos_plano_tau)

    #Se pinta
    fig, ax = plt.subplots(1,2)

    #coeficiente de presión
    fig.set_size_inches(15,15)

    ax[0].contourf(XX, YY, cp, 200, cmap=cmap)
    ax[0].grid()
    ax[0].set_aspect(1)
    p = plt.Polygon(list(zip(Xc, Yc)), color="#cccccc", zorder=10)
    ax[0].add_patch(p)
    ax[0].set_xlim(-5, 5)
    ax[0].set_ylim(-3,3)

    ax[1].contourf(XX_tau, YY_tau, cp, 200, cmap=cmap)
    ax[1].grid()
    ax[1].set_aspect(1)
    p = plt.Polygon(list(zip(Xp, Yp)), color="#cccccc", zorder=10)
    ax[1].add_patch(p)
    ax[1].set_xlim(-5, 5)
    ax[1].set_ylim(-3,3)
    plt.show()
```

```python
interact(cp_perfil_circunferencia, landa=(0.,1), delta=(0.,1), alfa=(-30, 30), U=(0,10), N_malla = (10,200))
```

    interactive(children=(FloatSlider(value=0.5, description='landa', max=1.0), FloatSlider(value=0.5, description…

    <function __main__.cp_perfil_circunferencia(landa, delta, alfa, U=1, N_malla=100)>

---

_En esta clase hemos reafirmado nuestros conocimientos de NumPy, matplotlib y Python, general (funciones, bucles, condicionales...) aplicándolos a un ejemplo muy aeronáutico_

Si te ha gustado esta clase:

Tweet

---

####
¡Síguenos en Twitter!

###### Follow @Pybonacci   Follow @Alex__S12   Follow @newlawrence

Curso AeroPython por Juan Luis Cano Rodriguez y Alejandro Sáez Mollejo se distribuye bajo una Licencia Creative Commons Atribución 4.0 Internacional.

---
_Las siguientes celdas contienen configuración del Notebook_

_Para visualizar y utlizar los enlaces a Twitter el notebook debe ejecutarse como [seguro](http://ipython.org/ipython-doc/dev/notebook/security.html)_

    File > Trusted Notebook

```python
%%html
<a href="https://twitter.com/Pybonacci" class="twitter-follow-button" data-show-count="false">Follow @Pybonacci</a>

```

Follow @Pybonacci

```python
# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = '../styles/aeropython.css'
HTML(open(css_file, "r").read())
```

/* This template is inspired in the one used by Lorena Barba
in the numerical-mooc repository: https://github.com/numerical-mooc/numerical-mooc
We thank her work and hope you also enjoy the look of the notobooks with this style */

<link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro|Josefin+Sans:400,700,400italic|Ubuntu+Condensed' rel='stylesheet' type='text/css'>

El estilo se ha aplicado =)


```python
from sympy import *
from sympy.abc import x,n
```

```python
for i in range(0,15,2):
    eq = x**i
    print(eq)
    plot(eq,diff(eq),diff(eq,x,2),legend=True,size = (15,5))
```

```python
plot(sin(x),diff(sin(x),yscale='log', size=(15,10),legend=True))
```

```python
from sympy import Sum
Sum(Derivative(x)+, (x, 1, oo))
```

$\displaystyle \sum_{x=1}^{\infty} \frac{d}{d x} x$

```python

```


```
%load_ext autoreload
```

```
autoreload 2
```

```
import matplotlib.pyplot as plt
import numpy as np
import sympy as sym

import inputs
import models
import shooting

from __future__ import division
```

```
# define some workers skill
x, a, b = sym.var('x, mu, sigma')
skill_cdf = 0.5 + 0.5 * sym.erf((sym.log(x) - mu) / (sym.sqrt(2) * sigma))
skill_params = {'mu': 0.0, 'sigma': 1.0}
skill_bounds = [1e-3, 2.0]

workers = inputs.Input(var=x,
                       cdf=skill_cdf,
                       params=skill_params,
                       bounds=skill_bounds,
                       )

# define some firms lognorm.pdf(x, s) = 1 / (s*x*sqrt(2*pi)) * exp(-1/2*(log(x)/s)**2)
y, mu2, sigma2 = sym.var('y, mu2, sigma2')
productivity_cdf = 0.5 + 0.5 * sym.erf((sym.log(y) - mu2) / (sym.sqrt(2) * sigma2))
productivity_params = {'mu2': 0.0, 'sigma2': 1.0}
productivity_bounds = [1e-3, 2.0]

firms = inputs.Input(var=y,
                     cdf=productivity_cdf,
                     params=productivity_params,
                     bounds=productivity_bounds,
                     )
```

```
# Developped country

R, l, r, rho, gamma, eta, L, A, kapa = sym.var('R, l, r, rho, gamma, eta, L, A, kapa')

# First set k to root 3
k_star = (2*6**(-0.5)*(A*eta**2*kapa*x**0.5/R)**0.5*sym.cos(0.333333333333333*sym.acos(3*6**0.5*x**(-0.5)*(l*x*y/r)**0.25*(R*x**(-0.5)/(A*eta**2*kapa))**0.5*(-eta + 1)/(2*eta))))**(4)

F = A*kapa*(eta*(k_star*x)**rho + (1- eta)*(y*(l/r))**rho)**(gamma/rho)

F_params = {'R':0.13099, 'rho':0.25, 'gamma':0.5, 'eta':0.89, 'L':4.2, 'A': 1.0, 'kapa':1.0}
```

```
# Try out positive
model = models.Model('positive',
                     workers=workers,
                     firms=firms,
                     production=F,
                     params=F_params)

solver = shooting.ShootingSolver(model=model)
```

```
solver.solve(2.0, tol=1e-6, number_knots=100, integrator='vode', atol=1e-15, rtol=1e-12, message=True)
```

```
solver.solution
```

      $\mu(x)$
      $\theta(x)$
      $w(x)$
      $\pi(x)$

      x

      0.240406
       1.660972
       29.918135
       0.007603
       1.137293

      0.243303
       1.660978
       29.498637
       0.007711
       1.145026

      0.263495
       1.661032
       26.817661
       0.008482
       1.198627

      0.283687
       1.661101
       24.503273
       0.009283
       1.251755

      0.303879
       1.661187
       22.488860
       0.010114
       1.304476

      0.324071
       1.661293
       20.722766
       0.010976
       1.356843

      0.344263
       1.661421
       19.164329
       0.011869
       1.408900

      0.364455
       1.661574
       17.781119
       0.012792
       1.460683

      0.384646
       1.661754
       16.546970
       0.013746
       1.512223

      0.404838
       1.661965
       15.440548
       0.014731
       1.563545

      0.425030
       1.662209
       14.444297
       0.015747
       1.614673

      0.445222
       1.662489
       13.543647
       0.016794
       1.665626

      0.465414
       1.662807
       12.726409
       0.017873
       1.716421

      0.485606
       1.663167
       11.982322
       0.018983
       1.767071

      0.505798
       1.663572
       11.302682
       0.020124
       1.817590

      0.525990
       1.664024
       10.680070
       0.021297
       1.867989

      0.546182
       1.664525
       10.108122
       0.022503
       1.918279

      0.566374
       1.665080
        9.581354
       0.023740
       1.968467

      0.586566
       1.665691
        9.095017
       0.025009
       2.018563

      0.606758
       1.666361
        8.644979
       0.026311
       2.068572

      0.626949
       1.667092
        8.227631
       0.027646
       2.118502

      0.647141
       1.667888
        7.839809
       0.029013
       2.168358

      0.667333
       1.668750
        7.478727
       0.030414
       2.218146

      0.687525
       1.669683
        7.141926
       0.031848
       2.267869

      0.707717
       1.670689
        6.827227
       0.033316
       2.317533

      0.727909
       1.671770
        6.532692
       0.034819
       2.367142

      0.748101
       1.672930
        6.256595
       0.036355
       2.416698

      0.768293
       1.674170
        5.997392
       0.037926
       2.466205

      0.788485
       1.675494
        5.753700
       0.039533
       2.515666

      0.808677
       1.676904
        5.524277
       0.041174
       2.565084

      ...
      ...
      ...
      ...
      ...

      1.414434
       1.771461
        2.074702
       0.109634
       4.035518

      1.434626
       1.776744
        2.018635
       0.112679
       4.084287

      1.454818
       1.782189
        1.964508
       0.115784
       4.133046

      1.475010
       1.787799
        1.912230
       0.118949
       4.181795

      1.495202
       1.793577
        1.861717
       0.122177
       4.230535

      1.515394
       1.799524
        1.812890
       0.125467
       4.279267

      1.535586
       1.805644
        1.765672
       0.128823
       4.327990

      1.555778
       1.811939
        1.719993
       0.132244
       4.376705

      1.575970
       1.818411
        1.675786
       0.135733
       4.425412

      1.596162
       1.825063
        1.632987
       0.139290
       4.474111

      1.616354
       1.831898
        1.591537
       0.142918
       4.522803

      1.636545
       1.838918
        1.551379
       0.146617
       4.571488

      1.656737
       1.846125
        1.512460
       0.150390
       4.620166

      1.676929
       1.853524
        1.474728
       0.154238
       4.668837

      1.697121
       1.861115
        1.438137
       0.158162
       4.717502

      1.717313
       1.868902
        1.402641
       0.162165
       4.766160

      1.737505
       1.876887
        1.368196
       0.166247
       4.814813

      1.757697
       1.885074
        1.334762
       0.170411
       4.863459

      1.777889
       1.893465
        1.302299
       0.174659
       4.912100

      1.798081
       1.902062
        1.270772
       0.178992
       4.960735

      1.818273
       1.910869
        1.240145
       0.183413
       5.009364

      1.838465
       1.919888
        1.210384
       0.187923
       5.057989

      1.858657
       1.929121
        1.181459
       0.192524
       5.106608

      1.878848
       1.938572
        1.153338
       0.197218
       5.155222

      1.899040
       1.948244
        1.125993
       0.202007
       5.203832

      1.919232
       1.958139
        1.099398
       0.206894
       5.252436

      1.939424
       1.968259
        1.073525
       0.211880
       5.301037

      1.959616
       1.978607
        1.048350
       0.216968
       5.349633

      1.979808
       1.989187
        1.023849
       0.222160
       5.398224

      2.000000
       2.000000
        1.000000
       0.227459
       5.446811

89 rows × 4 columns

```
solver.residuals.plot(logy=True, alpha=0.5)
plt.show()
```

```
solver.residuals
```

```

```


# Logistic Regression

    Notebook version: 1.0 (Oct 12, 2016)

    Author: Jesús Cid Sueiro (jcid@tsc.uc3m.es)
            Jerónimo Arenas García (jarenas@tsc.uc3m.es)

    Changes: v.1.0 - First version
             v.1.1 - Typo correction. Prepared for slide presentation

```python
# To visualize plots in the notebook
%matplotlib inline

# Imported libraries
import csv
import random
import matplotlib
import matplotlib.pyplot as plt
import pylab

import numpy as np
from mpl_toolkits.mplot3d import Axes3D
from sklearn.preprocessing import PolynomialFeatures
from sklearn import linear_model

```

    /Users/jcid/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

# Logistic Regression

## 1. Introduction

### 1.1. Binary classification and decision theory. The MAP criterion

Goal of a classification problem is to assign a *class* or *category* to every *instance* or *observation* of a data collection. Here, we will assume that every instance ${\bf x}$ is an $N$-dimensional vector in $\mathbb{R}^N$, and that the class $y$ of sample ${\bf x}$ is an element of a binary set ${\mathcal Y} = \{0, 1\}$. The goal of a classifier is to predict the true value of $y$ after observing ${\bf x}$.

We will denote as $\hat{y}$ the classifier output or *decision*. If $y=\hat{y}$, the decision is an *hit*, otherwise $y\neq \hat{y}$ and the decision is an *error*.

Decision theory provides a solution to the classification problem in situations where the relation between instance ${\bf x}$ and its class $y$ is given by a known probabilistic model: assume that every tuple $({\bf x}, y)$ is an outcome of a random vector $({\bf X}, Y)$ with joint distribution $p_{{\bf X},Y}({\bf x}, y)$. A natural criteria for classification is to select predictor $\hat{Y}=f({\bf x})$ in such a way that the probability or error, $P\{\hat{Y} \neq Y\}$ is minimum. Noting that

$$
P\{\hat{Y} \neq Y\} = \int P\{\hat{Y} \neq Y | {\bf x}\} p_{\bf X}({\bf x}) d{\bf x}
$$

the optimal decision is got if, for every sample ${\bf x}$, we make decision minimizing the conditional error probability:

\begin{align}
\hat{y}^* &= \arg\min_{\hat{y}} P\{\hat{y} \neq Y |{\bf x}\} \\
          &= \arg\max_{\hat{y}} P\{\hat{y} = Y |{\bf x}\} \\
\end{align}

Thus, the optimal decision rule can be expressed as

$$
P_{Y|{\bf X}}(1|{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  P_{Y|{\bf X}}(0|{\bf x})
$$

or, equivalently

$$
P_{Y|{\bf X}}(1|{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  \frac{1}{2}
$$

The classifier implementing this decision rule is usually named MAP (*Maximum A Posteriori*).

### 1.2. Parametric classification.

Classical decision theory is grounded on the assumption that the probabilistic model relating the observed sample ${\bf X}$ and the true hypothesis $Y$ is known. Unfortunately, this is unrealistic in many applications, where the only available information to construct the classifier is a dataset $\mathcal S = \{({\bf x}^{(k)}, y^{(k)}), \,k=1,\ldots,K\}$ of instances and their respective class labels.

A more realistic formulation of the classification problem is the following: given a dataset $\mathcal S = \{({\bf x}^{(k)}, y^{(k)}) \in {\mathbb{R}}^N \times {\mathcal Y}, \, k=1,\ldots,K\}$ of independent and identically distributed (i.i.d.) samples from an ***unknown*** distribution $p_{{\bf X},Y}({\bf x}, y)$, predict the class $y$ of a new sample ${\bf x}$ with the minimum probability of error.

Since the probabilistic model generating the data is unknown, the MAP decision rule cannot be applied. However, many classification algorithms use the dataset to obtain an estimate of the posterior class probabilities, and apply it to implement an approximation to the MAP decision maker.

Parametric classifiers based on this idea assume, additionally, that the posterior class probabilty satisfies some parametric formula:

$$
P_{Y|X}(1|{\bf x},{\bf w}) = f_{\bf w}({\bf x})
$$

where ${\bf w}$ is a vector of parameters. Given the expression of the MAP decision maker, classification consists in comparing the value of $f_{\bf w}({\bf x})$ with the threshold $\frac{1}{2}$, and each parameter vector would be associated to a different decision maker.

In practice, the dataset ${\mathcal S}$ is used to select a particular parameter vector $\hat{\bf w}$ according to certain criterion. Accordingly, the decision rule becomes

$$
f_{\hat{\bf w}}({\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad  \frac{1}{2}
$$

In this lesson, we explore one of the most popular model-based parametric classification methods: **logistic regression**.

## 2. Logistic regression.

### 2.1. The logistic function

The logistic regression model assumes that the binary class label $Y \in \{0,1\}$ of observation $X\in \mathbb{R}^N$ satisfies the expression.

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g({\bf w}^\intercal{\bf x})$$
$$P_{Y|{\bf,X}}(0|{\bf x}, {\bf w}) = 1-g({\bf w}^\intercal{\bf x})$$

where ${\bf w}$ is a parameter vector and $g(·)$ is the *logistic* function, which is defined by

$$g(t) = \frac{1}{1+\exp(-t)}$$

It is straightforward to see that the logistic function has the following properties:

- **P1**: Probabilistic output: $\quad 0 \le g(t) \le 1$
- **P2**: Symmetry: $\quad g(-t) = 1-g(t)$
- **P3**: Monotonicity: $\quad g'(t) = g(t)·[1-g(t)] \ge 0$

In the following we define a logistic function in python, and use it to plot a graphical representation.

**Exercise 1**: Verify properties P2 and P3.

**Exercise 2**: Implement a function to compute the logistic function, and use it to plot such function in the inverval $[-6,6]$.

```python
# Define the logistic function
def logistic(x):
   p = 1.0 / (1 + np.exp(-x))
   return p

# Plot the logistic function
t = np.arange(-6, 6, 0.1)
z = logistic(t)

plt.plot(t, z)
plt.xlabel('$t$', fontsize=14)
plt.ylabel('$\phi(t)$', fontsize=14)
plt.title('The logistic function')
plt.grid()
```

### 2.2. Classifiers based on the logistic model.

The MAP classifier under a logistic model will have the form

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g({\bf w}^\intercal{\bf x}) \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad \frac{1}{2} $$

Therefore

$$
2 \quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0} \quad
1 + \exp(-{\bf w}^\intercal{\bf x}) $$

which is equivalent to

$${\bf w}^\intercal{\bf x}
\quad\mathop{\gtrless}^{\hat{y}=1}_{\hat{y}=0}\quad
0 $$

Therefore, the classifiers based on the logistic model are given by linear decision boundaries passing through the origin, ${\bf x} = {\bf 0}$.

```python
# Weight vector:
w = [1, 4, 8]   # Try different weights

# Create a rectangular grid.
x_min = -1
x_max = 1
dx = x_max - x_min
h = float(dx) / 200
xgrid = np.arange(x_min, x_max, h)
xx0, xx1 = np.meshgrid(xgrid, xgrid)

# Compute the logistic map for the given weights
Z = logistic(w[0] + w[1]*xx0 + w[2]*xx1)

# Plot the logistic map
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(xx0, xx1, Z, cmap=plt.cm.copper)
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
ax.set_zlabel('P(1|x,w)')
```

### 3.3. Nonlinear classifiers.

The logistic model can be extended to construct non-linear classifiers by using non-linear data transformations. A general form for a nonlinear logistic regression model is

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g[{\bf w}^\intercal{\bf z}({\bf x})] $$

where ${\bf z}({\bf x})$ is an arbitrary nonlinear transformation of the original variables. The boundary decision in that case is given by equation

$$
{\bf w}^\intercal{\bf z} = 0
$$

** Exercise 2**: Modify the code above to generate a 3D surface plot of the polynomial logistic regression model given by

$$
P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g(1 + 10 x_0 + 10 x_1 - 20 x_0^2 + 5 x_0 x_1 + x_1^2)
$$

```python
# SOLUTION TO THE EXERCISE
# Weight vector:
w = [1, 10, 10, -20, 5, 1]   # Try different weights

# Create a regtangular grid.
x_min = -1
x_max = 1
dx = x_max - x_min
h = float(dx) / 200
xgrid = np.arange(x_min, x_max, h)
xx0, xx1 = np.meshgrid(xgrid, xgrid)

# Compute the logistic map for the given weights
Z = logistic(w[0] + w[1]*xx0 + w[2]*xx1 + w[3]*np.multiply(xx0,xx0) +
             w[4]*np.multiply(xx0,xx1) + w[3]*np.multiply(xx1,xx1))

# Plot the logistic map
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(xx0, xx1, Z, cmap=plt.cm.copper)
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
ax.set_zlabel('P(1|x,w)')
```

## 3. Inference

Remember that the idea of parametric classification is to use the training data set $\mathcal S = \{({\bf x}^{(k)}, y^{(k)}) \in {\mathbb{R}}^N \times \{0,1\}, k=1,\ldots,K\}$ to set the parameter vector ${\bf w}$ according to certain criterion. Then, the estimate $\hat{\bf w}$ can be used to compute the label prediction for any new observation as

$$\hat{y} = \arg\max_y P_{Y|{\bf X}}(y|{\bf x},\hat{\bf w}).$$

In the following, we will make the following assumptions:

- **A1**. The samples in ${\mathcal S}$ are i.i.d.

- **A2**. Target $Y^{(k)}$ only depends on ${\bf x}^{(k)}$, but not on ${\bf x}^{(l)}$ for any $l\neq k$.

- **A3**. (Logistic Regression): We assume a logistic model for the *a posteriori* probability of ${Y=1}$ given ${\bf X}$, i.e.,

$$P_{Y|{\bf X}}(1|{\bf x}, {\bf w}) = g[{\bf w}^\intercal{\bf z}({\bf x})].$$

We need still to choose a criterion to optimize with the selection of the parameter vector. In the notebook, we will discuss two different approaches to the estimation of ${\bf w}$:

   * Maximum Likelihood (ML): $\hat{\bf w}_{\text{ML}} = \arg\max_{\bf w} P_{{\mathcal S}|{\bf W}}({\mathcal S}|{\bf w})$
   * Maximum *A Posteriori* (MAP): $\hat{\bf w}_{\text{MAP}} = \arg\max_{\bf w} p_{{\bf W}|{\mathcal S}}({\bf w}|{\mathcal S})$

For the mathematical derivation of the logistic regression algorithm, the following representation of the logistic model will be useful: noting that

$$P_{Y|{\bf X}}(0|{\bf x}, {\bf w}) = 1-g[{\bf w}^\intercal{\bf z}({\bf x})]
= g[-{\bf w}^\intercal{\bf z}({\bf x})]$$

we can write

$$P_{Y|{\bf X}}(y|{\bf x}, {\bf w}) = g[\overline{y}{\bf w}^\intercal{\bf z}({\bf x})]$$

where $\overline{y} = 2y-1$ is a *symmetrized label* ($\overline{y}\in\{-1, 1\}$).

### 3.1. ML estimation.

The ML estimate is defined as

$$\hat{\bf w}_{\text{ML}} = \arg\max_{\bf w} P_{{\mathcal S}|{\bf W}}({\mathcal S}|{\bf w})
   = \arg\min_{\bf w} L({\bf w})
$$

where $L({\bf w})$ is the negative log-likelihood function, given by

$$
L({\bf w}) = - \log P_{{\mathcal S}|{\bf W}}({\mathcal S}|{\bf w})
 = - \log\left[P\left(y^{(1)},\ldots,y^{(K)}|
     {\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)\right]
$$

Using assumption A1,

$$
L({\bf w}) = - \log\left[\prod_{k=1}^K P\left(y^{(k)}|{\bf x}^{(1)},\ldots,{\bf x}^{(K)},{\bf w}\right)\right].
$$

Using A2,

\begin{align}
L({\bf w})
    &= - \log\left[\prod_{k=1}^K P_{Y|{\bf X}}\left(y^{(k)}|{\bf x}^{(k)},{\bf w}\right)\right] \\
    &= - \sum_{k=1}^K\log\left[P_{Y|{\bf X}}\left(y^{(k)}|{\bf x}^{(k)},{\bf w}\right)\right]
\end{align}

Using A3 (the logistic model)

\begin{align}
L({\bf w})
    &= - \sum_{k=1}^K\log\left[g\left(\overline{y}^{(k)}{\bf w}^\intercal {\bf z}^{(k)}\right)\right]   \\
    &= \sum_{k=1}^K\log\left[1+\exp\left(-\overline{y}^{(k)}{\bf w}^\intercal {\bf z}^{(k)}\right)\right]
\end{align}

where ${\bf z}^{(k)}={\bf z}({\bf x}^{(k)})$.

It can be shown that $L({\bf w})$ is a convex and differentiable function of ${\bf w}$. Therefore, its minimum is a point with zero gradient.

\begin{align}
\nabla_{\bf w} L(\hat{\bf w}_{\text{ML}})
    &= - \sum_{k=1}^K
       \frac{\exp\left(-\overline{y}^{(k)}\hat{\bf w}_{\text{ML}}^\intercal {\bf z}^{(k)}\right) \overline{y}^{(k)} {\bf z}^{(k)}}
       {1+\exp\left(-\overline{y}^{(k)}\hat{\bf w}_{\text{ML}}^\intercal {\bf z}^{(k)}
       \right)} = \\
    &= - \sum_{k=1}^K \left[y^{(k)}-g(\hat{\bf w}_{\text{ML}}^T {\bf z}^{(k)})\right] {\bf z}^{(k)} = 0
\end{align}

Unfortunately, $\hat{\bf w}_{\text{ML}}$ cannot be taken out from the above equation, and some iterative optimization algorithm must be used to search for the minimum.

### 3.2. Gradient descent.

A simple iterative optimization algorithm is  gradient descent.

\begin{align}
{\bf w}_{n+1} = {\bf w}_n - \rho_n \nabla_{\bf w} L({\bf w}_n)
\end{align}

where $\rho_n >0$ is the *learning step*.

Applying the gradient descent rule to logistic regression, we get the following algorithm:

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n \sum_{k=1}^K \left[y^{(k)}-g({\bf w}_n^\intercal {\bf z}^{(k)})\right] {\bf z}^{(k)}
\end{align}

Defining vectors

\begin{align}
{\bf y} &= [y^{(1)},\ldots,y^{(K)}]^\intercal \\
\hat{\bf p}_n &= [g({\bf w}_n^\intercal {\bf z}^{(1)}), \ldots, g({\bf w}_n^\intercal {\bf z}^{(K)})]^\intercal
\end{align}
and matrix
\begin{align}
{\bf Z} = \left[{\bf z}^{(1)},\ldots,{\bf z}^{(K)}\right]^\intercal
\end{align}

we can write

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n {\bf Z} \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

In the following, we will explore the behavior of the gradient descend method using the Iris Dataset.

#### 3.2.1 Example: Iris Dataset.

As an illustration, consider the  Iris dataset , taken from the  UCI Machine Learning repository. This data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant (*setosa*, *versicolor* or *virginica*). Each instance contains 4 measurements of given flowers: sepal length, sepal width, petal length and petal width, all in centimeters.

We will try to fit the logistic regression model to discriminate between two classes using only two attributes.

First, we load the dataset and split them in training and test subsets.

```python
# Adapted from a notebook by Jason Brownlee
def loadDataset(filename, split):
    xTrain = []
    cTrain = []
    xTest = []
    cTest = []

    with open(filename, 'rb') as csvfile:
        lines = csv.reader(csvfile)
        dataset = list(lines)
    for i in range(len(dataset)-1):
        for y in range(4):
            dataset[i][y] = float(dataset[i][y])
        item = dataset[i]
        if random.random() < split:
            xTrain.append(item[0:4])
            cTrain.append(item[4])
        else:
            xTest.append(item[0:4])
            cTest.append(item[4])
    return xTrain, cTrain, xTest, cTest

with open('iris.data', 'rb') as csvfile:
    lines = csv.reader(csvfile)

xTrain_all, cTrain_all, xTest_all, cTest_all = loadDataset('iris.data', 0.66)
nTrain_all = len(xTrain_all)
nTest_all = len(xTest_all)
print 'Train: ' + str(nTrain_all)
print 'Test: ' + str(nTest_all)
```

    Train: 95
    Test: 55

Now, we select two classes and two attributes.

```python
# Select attributes
i = 0 # Try 0,1,2,3
j = 1 # Try 0,1,2,3 with j!=i

# Select two classes
c0 = 'Iris-versicolor'
c1 = 'Iris-virginica'

# Select two coordinates
ind = [i, j]

# Take training test
X_tr = np.array([[xTrain_all[n][i] for i in ind] for n in range(nTrain_all)
                  if cTrain_all[n]==c0 or cTrain_all[n]==c1])
C_tr = [cTrain_all[n] for n in range(nTrain_all)
          if cTrain_all[n]==c0 or cTrain_all[n]==c1]
Y_tr = np.array([int(c==c1) for c in C_tr])
n_tr = len(X_tr)

# Take test set
X_tst = np.array([[xTest_all[n][i] for i in ind] for n in range(nTest_all)
                 if cTest_all[n]==c0 or cTest_all[n]==c1])
C_tst = [cTest_all[n] for n in range(nTest_all)
         if cTest_all[n]==c0 or cTest_all[n]==c1]
Y_tst = np.array([int(c==c1) for c in C_tst])
n_tst = len(X_tst)
```

#### 3.2.2. Data normalization

Normalization of data is a common pre-processing step in many machine learning algorithms. Its goal is to get a dataset where all input coordinates have a similar scale. Learning algorithms usually show less instabilities and convergence problems when data are normalized.

We will define a normalization function that returns a training data matrix with zero sample mean and unit sample variance.

```python
def normalize(X, mx=None, sx=None):

    # Compute means and standard deviations
    if mx is None:
        mx = np.mean(X, axis=0)
    if sx is None:
        sx = np.std(X, axis=0)

    # Normalize
    X0 = (X-mx)/sx

    return X0, mx, sx
```

Now, we can normalize training and test data. Observe in the code that the same transformation should be applied to training and test data. This is the reason why normalization with the test data is done using the means and the variances computed with the training set.

```python
# Normalize data
Xn_tr, mx, sx = normalize(X_tr)
Xn_tst, mx, sx = normalize(X_tst, mx, sx)
```

The following figure generates a plot of the normalized training data.

```python
# Separate components of x into different arrays (just for the plots)
x0c0 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==0]
x1c0 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==0]
x0c1 = [Xn_tr[n][0] for n in range(n_tr) if Y_tr[n]==1]
x1c1 = [Xn_tr[n][1] for n in range(n_tr) if Y_tr[n]==1]

# Scatterplot.
labels = {'Iris-setosa': 'Setosa',
          'Iris-versicolor': 'Versicolor',
          'Iris-virginica': 'Virginica'}
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')
```

In order to apply the gradient descent rule, we need to define two methods:
 - A `fit` method, that receives the training data and returns the model weights and the value of the negative log-likelihood during all iterations.
 - A `predict` method, that receives the model weight and a set of inputs, and returns the posterior class probabilities for that input, as well as their corresponding class predictions.

```python
def logregFit(Z_tr, Y_tr, rho, n_it):

    # Data dimension
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    nll_tr2 = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):

        # Compute posterior probabilities for weight w
        p1_tr = logistic(np.dot(Z_tr, w))

        # Compute negative log-likelihood
        # (note that this is not required for the weight update, only for nll tracking)
        Y_tr2 = 2*Y_tr - 1
        nll_tr[n] = np.sum(np.log(1 + np.exp(-np.dot(Y_tr2*Z_tr, w))))

        # Update weights
        w += rho*np.dot(Z_tr.T, Y_tr - p1_tr)

    return w, nll_tr

def logregPredict(Z, w):

    # Compute posterior probability of class 1 for weights w.
    p = logistic(np.dot(Z, w))

    # Class
    D = [int(round(pn)) for pn in p]

    return p, D
```

We can test the behavior of the gradient descent method by fitting a logistic regression model with ${\bf z}({\bf x}) = (1, {\bf x}^\intercal)^\intercal$.

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 200   # Number of iterations

# Compute Z's
Z_tr = np.c_[np.ones(n_tr), Xn_tr]
Z_tst = np.c_[np.ones(n_tst), Xn_tst]
n_dim = Z_tr.shape[1]

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The optimal weights are:"
print w
print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])
```

#### 3.2.3. Free parameters

Under certain conditions, the gradient descent method can be shown to converge asymptotically (i.e. as the number of iterations goes to infinity) to the ML estimate of the logistic model. However, in practice, the final estimate of the weights ${\bf w}$ depend on several factors:

- Number of iterations
- Initialization
- Learning step

**Exercise**: Visualize the variability of gradient descent caused by initializations. To do so, fix the number of iterations to 200 and the learning step, and execute the gradient descent 100 times, storing the training error rate of each execution. Plot the histogram of the error rate values.

Note that you can do this exercise with a loop over the 100 executions, including the code in the previous code slide inside the loop, with some proper modifications. To plot a histogram of the values in array `p` with `n`bins, you can use `plt.hist(p, n)`

##### 3.2.3.1. Learning step

The learning step, $\rho$, is a free parameter of the algorithm. Its choice is critical for the convergence of the algorithm. Too large values of $\rho$ make the algorithm diverge. For too small values, the convergence gets very slow and more iterations are required for a good convergence.

**Exercise 3**: Observe the evolution of the negative log-likelihood with the number of iterations for different values of $\rho$. It is easy to check that, for large enough $\rho$, the gradient descent method does not converge. Can you estimate (through manual observation) an approximate value of $\rho$ stating a boundary between convergence and divergence?

**Exercise 4**: In this exercise we explore the influence of the learning step more sistematically. Use the code in the previouse exercises to compute, for every value of $\rho$, the average error rate over 100 executions. Plot the average error rate vs. $\rho$.

Note that you should explore the values of $\rho$ in a logarithmic scale. For instance, you can take $\rho = 1, 1/10, 1/100, 1/1000, \ldots$

In practice, the selection of $\rho$ may be a matter of trial an error. Also there is some theoretical evidence that the learning step should decrease along time up to cero, and the sequence $\rho_n$ should satisfy two conditions:
- C1: $\sum_{n=0}^{\infty} \rho_n^2 < \infty$ (decrease slowly)
- C2: $\sum_{n=0}^{\infty} \rho_n = \infty$ (but not too slowly)

For instance, we can take $\rho_n= 1/n$. Another common choice is $\rho_n = \alpha/(1+\beta n)$ where $\alpha$ and $\beta$ are also free parameters that can be selected by trial and error with some heuristic method.

#### 3.2.4. Visualizing the posterior map.

We can also visualize the posterior probability map estimated by the logistic regression model for the estimated weights.

```python
# Create a regtangular grid.
x_min, x_max = Xn_tr[:, 0].min(), Xn_tr[:, 0].max()
y_min, y_max = Xn_tr[:, 1].min(), Xn_tr[:, 1].max()
dx = x_max - x_min
dy = y_max - y_min
h = dy /400
xx, yy = np.meshgrid(np.arange(x_min - 0.1 * dx, x_max + 0.1 * dx, h),
                     np.arange(y_min - 0.1 * dx, y_max + 0.1 * dy, h))
X_grid = np.array([xx.ravel(), yy.ravel()]).T

# Compute Z's
Z_grid = np.c_[np.ones(X_grid.shape[0]), X_grid]

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)

# Put the result into a color plot
plt.plot(x0c0, x1c0,'r.', label=labels[c0])
plt.plot(x0c1, x1c1,'g+', label=labels[c1])
plt.xlabel('$x_' + str(ind[0]) + '$')
plt.ylabel('$x_' + str(ind[1]) + '$')
plt.legend(loc='best')
plt.axis('equal')
pp = pp.reshape(xx.shape)
plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
```

#### 3.2.5. Polynomial Logistic Regression

The error rates of the logistic regression model can be potentially reduced by using polynomial transformations.

To compute the polynomial transformation up to a given degree, we can use the `PolynomialFeatures` method in `sklearn.preprocessing`.

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
g = 5 # Degree of polynomial

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit(Z_tr, Y_tr2, rho, n_it)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The optimal weights are:"
print w
print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])

```

Visualizing the posterior map we can se that the polynomial transformation produces nonlinear decision boundaries.

```python
# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
n_grid = Z_grid.shape[0]
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)

# Compute the classifier output for all samples in the grid.
pp, dd = logregPredict(Z_grid, w)
pp = pp.reshape(xx.shape)

# Paint output maps
pylab.rcParams['figure.figsize'] = 8, 4  # Set figure size
for i in [1, 2]:
    ax = plt.subplot(1,2,i)
    ax.plot(x0c0, x1c0,'r.', label=labels[c0])
    ax.plot(x0c1, x1c1,'g+', label=labels[c1])
    ax.set_xlabel('$x_' + str(ind[0]) + '$')
    ax.set_ylabel('$x_' + str(ind[1]) + '$')
    ax.axis('equal')
    if i==1:
        ax.contourf(xx, yy, pp, cmap=plt.cm.copper)
    else:
        ax.legend(loc='best')
        ax.contourf(xx, yy, np.round(pp), cmap=plt.cm.copper)
```

## 4. Regularization and MAP estimation.

An alternative to the ML estimation of the weights in logistic regression is Maximum A Posteriori estimation. Modelling the logistic regression weights as a random variable with prior distribution $p_{\bf W}({\bf w})$, the MAP estimate is defined as

$$
\hat{\bf w}_{\text{MAP}} = \arg\max_{\bf w} p({\bf w}|{\mathcal S})
$$

The posterior density $p({\bf w}|{\mathcal S})$ is related to the likelihood function and the prior density of the weights, $p_{\bf W}({\bf w})$ through the Bayes rule

$$
p({\bf w}|{\mathcal S}) =
    \frac{P\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)
          p_{\bf W}({\bf w})}
         {p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)}\right)}
$$

$$
p({\bf w}|{\mathcal S}) =
    \frac{P\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)
          p_{\bf W}({\bf w})}
         {p\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)}\right)}
$$

The numerator of the above expression is the product of two terms:

   - The likelihood $P_{{\mathcal S}|{\bf W}}({\mathcal S}|{\bf w})$, which takes large values for parameter vectors $\bf w$ that fit well the training data
   - The prior distribution of weights $p_{\bf W}({\bf w})$, which expresses our *a priori* preference for some solutions. Usually, we recur to prior distributions that take large values when $\|{\bf w}\|$ is small (associated to soft classification borders).

In general, the denominator in this expression cannot be computed analytically. However, it is not required for MAP estimation because it does not depend on ${\bf w}$.

Therefore, the MAP criterion prefers solutions that simultaneously fit well the data and our *a priori* belief about which solutions should be preferred.

$$\hat{\bf w}_{\text{MAP}}
  = \arg\max_{\bf w} P_{{\mathcal S}|{\bf W}}({\mathcal S}|{\bf w}) \cdot p_{\bf W}({\bf w})$$

We can compute the MAP estimate as

\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\max_{\bf w}
         P\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right)
         p_{\bf W}({\bf w}) \\
  &= \arg\max_{\bf w} \left\{
         \log\left[P\left(y^{(1)},\ldots,y^{(K)}|{\bf x}^{(1)},\ldots, {\bf x}^{(K)},{\bf w}\right) \right]
         + \log\left[ p_{\bf W}({\bf w})\right]
         \right\}  \\
  &= \arg\min_{\bf w} \left\{L({\bf w}) - \log\left[ p_{\bf W}({\bf w})\right]
         \right\}
\end{align}

where $L(·)$ is the negative log-likelihood function.

We can check that the MAP criterion adds a penalty term to the ML objective, that penalizes parameter vectors for which the prior distribution of weights takes small values.

### 4.1 MAP estimation with Gaussian prior

If we assume that ${\bf W}$ is a zero-mean Gaussian random variable with variance matrix $v{\bf I}$,

$$
p_{\bf W}({\bf w}) = \frac{1}{(2\pi v)^{N/2}} \exp\left(-\frac{1}{2v}\|{\bf w}\|^2\right)
$$

the MAP estimate becomes

\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\min_{\bf w} \left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|^2
         \right\}
\end{align}

where $C = 2v$. Noting that

$$\nabla_{\bf w}\left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|^2\right\}
= - {\bf Z} \left({\bf y}-\hat{\bf p}_n\right) + \frac{2}{C}{\bf w},
$$

we obtain the following gradient descent rule for MAP estimation

\begin{align}
{\bf w}_{n+1} &= \left(1-\frac{2\rho_n}{C}\right){\bf w}_n
    + \rho_n {\bf Z} \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

### 4.2 MAP estimation with Laplacian prior

If we assume that ${\bf W}$ follows a multivariate zero-mean Laplacian distribution given by

$$
p_{\bf W}({\bf w}) = \frac{1}{(2 C)^{N}} \exp\left(-\frac{1}{C}\|{\bf w}\|_1\right)
$$

(where $\|{\bf w}\|=|w_1|+\ldots+|w_N|$ is the $L_1$ norm of ${\bf w}$), the MAP estimate is

\begin{align}
\hat{\bf w}_{\text{MAP}}
  &= \arg\min_{\bf w} \left\{L({\bf w}) + \frac{1}{C}\|{\bf w}\|_1
         \right\}
\end{align}

The additional term introduced by the prior in the optimization algorithm is usually named the *regularization term*. It is usually very effective to avoid overfitting when the dimension of the weight vectors is high. Parameter $C$ is named the *inverse regularization strength*.

**Exercise 5**: Derive the gradient descent rules for MAP estimation of the logistic regression weights with Laplacian prior.

## 5. Other optimization algorithms

### 5.1. Stochastic Gradient descent.

Stochastic gradient descent (SGD) is based on the idea of using a single sample at each iteration of the learning algorithm. The SGD rule for ML logistic regression is

\begin{align}
{\bf w}_{n+1} &= {\bf w}_n
    + \rho_n {\bf z}^{(n)} \left(y^{(n)}-\hat{p}^{(n)}_n\right)
\end{align}

Once all samples in the training set have been applied, the algorith can continue by applying the training set several times.

The computational cost of each iteration of SGD is much smaller than that of gradient descent, though it usually needs more iterations to converge.

**Exercise 5**: Modify logregFit to implement an algorithm that applies the SGD rule.

### 5.2. Newton's method

Assume that the function to be minimized, $C({\bf w})$, can be approximated by its second order Taylor series expansion around ${\bf w}_0$

$$
C({\bf w}) \approx C({\bf w}_0)
+ \nabla_{\bf w}^\intercal C({\bf w}_0)({\bf w}-{\bf w}_0)
+ \frac{1}{2}({\bf w}-{\bf w}_0)^\intercal{\bf H}({\bf w}_0)({\bf w}-{\bf w}_0)
$$

where ${\bf H}({\bf w}_k)$ is the  *Hessian* matrix of $C$ at ${\bf w}_k$. Taking the gradient of $C({\bf w})$, and setting the result to ${\bf 0}$, the minimum of C around ${\bf w}_0$ can be approximated as

$$
{\bf w}^* = {\bf w}_0 - {\bf H}({\bf w}_0)^{-1} \nabla_{\bf w}^\intercal C({\bf w}_0)
$$

Since the second order polynomial is only an approximation to $C$, ${\bf w}^*$ is only an approximation to the optimal weight vector, but we can expect ${\bf w}^*$ to be closer to the minimizer of $C$ than ${\bf w}_0$. Thus, we can repeat the process, computing a second order approximation around ${\bf w}^*$ and a new approximation to the minimizer.

 Newton's method is based on this idea. At each optization step, the function to be minimized is approximated by a second order approximation using a Taylor series expansion around the current estimate. As a result, the learning rules becomes

$$\hat{\bf w}_{n+1} = \hat{\bf w}_{n} - \rho_n {\bf H}({\bf w}_k)^{-1} \nabla_{{\bf w}}C({\bf w}_k)
$$

For instance, for the MAP estimate with Gaussian prior, the *Hessian* matrix becomes

$$
{\bf H}({\bf w})
  = \frac{2}{C}{\bf I} + \sum_{k=1}^K f({\bf w}^T {\bf z}^{(k)}) \left(1-f({\bf w}^T {\bf z}^{(k)})\right){\bf z}^{(k)} ({\bf z}^{(k)})^\intercal
$$

Defining diagonal matrix

$$
{\mathbf S}({\bf w}) = \text{diag}\left(f({\bf w}^T {\bf z}^{(k)}) \left(1-f({\bf w}^T {\bf z}^{(k)})\right)\right)
$$

the Hessian matrix can be written in more compact form as

$$
{\bf H}({\bf w})
  = \frac{2}{C}{\bf I} + {\bf Z}^\intercal {\bf S}({\bf w}) {\bf Z}
$$

Therefore, the Newton's algorithm for logistic regression becomes

\begin{align}
\hat{\bf w}_{n+1} = \hat{\bf w}_{n} +
\rho_n
\left(\frac{2}{C}{\bf I} + {\bf Z}^\intercal {\bf S}(\hat{\bf w}_{n})
{\bf Z}
\right)^{-1}
{\bf Z}^\intercal \left({\bf y}-\hat{\bf p}_n\right)
\end{align}

Some variants of the Newton method are implemented in the  Scikit-learn  package.

```python
def logregFit2(Z_tr, Y_tr, rho, n_it, C=1e4):

    # Compute Z's
    r = 2.0/C
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):
        p_tr = logistic(np.dot(Z_tr, w))

        sk = np.multiply(p_tr, 1-p_tr)
        S = np.diag(np.ravel(sk.T))

        # Compute negative log-likelihood
        nll_tr[n] = - np.dot(Y_tr.T, np.log(p_tr)) - np.dot((1-Y_tr).T, np.log(1-p_tr))

        # Update weights
        invH = np.linalg.inv(r*np.identity(n_dim) + np.dot(Z_tr.T, np.dot(S, Z_tr)))

        w += rho*np.dot(invH, np.dot(Z_tr.T, Y_tr - p_tr))

    return w, nll_tr
```

```python
# Parameters of the algorithms
rho = float(1)/50    # Learning step
n_it = 500   # Number of iterations
C = 1000
g = 4

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(X_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(X_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Convert target arrays to column vectors
Y_tr2 = Y_tr[np.newaxis].T
Y_tst2 = Y_tst[np.newaxis].T

# Running the gradient descent algorithm
w, nll_tr = logregFit2(Z_tr, Y_tr2, rho, n_it, C)

# Classify training and test data
p_tr, D_tr = logregPredict(Z_tr, w)
p_tst, D_tst = logregPredict(Z_tst, w)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

# NLL plot.
plt.plot(range(n_it), nll_tr,'b.:', label='Train')
plt.xlabel('Iteration')
plt.ylabel('Negative Log-Likelihood')
plt.legend()

print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)
print "The NLL after training is " + str(nll_tr[len(nll_tr)-1])
```

## 6. Logistic regression in Scikit Learn.

The  scikit-learn  package includes an efficient implementation of  logistic regression. To use it, we must first create a classifier object, specifying the parameters of the logistic regression algorithm.

```python
# Create a logistic regression object.
LogReg = linear_model.LogisticRegression(C=1.0)

# Compute Z_tr
poly = PolynomialFeatures(degree=g)
Z_tr = poly.fit_transform(Xn_tr)
# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr,1)), Zn), axis=1)

# Compute Z_tst
Z_tst = poly.fit_transform(Xn_tst)
Zn, mz, sz = normalize(Z_tst[:,1:], mz, sz)
Z_tst = np.concatenate((np.ones((n_tst,1)), Zn), axis=1)

# Fit model to data.
LogReg.fit(Z_tr, Y_tr)

# Classify training and test data
D_tr = LogReg.predict(Z_tr)
D_tst = LogReg.predict(Z_tst)

# Compute error rates
E_tr = D_tr!=Y_tr
E_tst = D_tst!=Y_tst

# Error rates
pe_tr = float(sum(E_tr)) / n_tr
pe_tst = float(sum(E_tst)) / n_tst

print "The final error rates are:"
print "- Training: " + str(pe_tr)
print "- Test: " + str(pe_tst)

# Compute Z_grid
Z_grid = poly.fit_transform(X_grid)
n_grid = Z_grid.shape[0]
Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)

# Compute the classifier output for all samples in the grid.
dd = LogReg.predict(Z_grid)
pp = LogReg.predict_proba(Z_grid)[:,1]
pp = pp.reshape(xx.shape)

# Paint output maps
pylab.rcParams['figure.figsize'] = 8, 4  # Set figure size
for i in [1, 2]:
    ax = plt.subplot(1,2,i)
    ax.plot(x0c0, x1c0,'r.', label=labels[c0])
    ax.plot(x0c1, x1c1,'g+', label=labels[c1])
    ax.set_xlabel('$x_' + str(ind[0]) + '$')
    ax.set_ylabel('$x_' + str(ind[1]) + '$')
    ax.axis('equal')
    if i==1:
        ax.contourf(xx, yy, pp, cmap=plt.cm.copper)
    else:
        ax.legend(loc='best')
        ax.contourf(xx, yy, np.round(pp), cmap=plt.cm.copper)

```


# Linear algebra overview

Linear algebra is the study of **vectors** and **linear transformations**. This notebook introduces concepts form linear algebra in a birds-eye overview. The goal is not to get into the details, but to give the reader a taste of the different types of thinking: computational, geometrical, and theoretical, that are used in linear algebra.

## Chapters overview

  - 1/ Math fundamentals
  - 2/ Intro to linear algebra
      - Vectors
      - Matrices
      - Matrix-vector product representation of linear transformations
      - Linear property: $f(a\mathbf{x} + b\mathbf{y}) = af(\mathbf{x}) + bf(\mathbf{y})$
  - 3/ Computational linear algebra
    - Gauss-Jordan elimination procedure
      - Augemnted matrix representaiton of systems of linear equations
      - Reduced row echelon form
    - Matrix equations
    - Matrix operations
      - Matrix product
      - Determinant
      - Matrix inverse
  - 4/ Geometrical linear algebra
    - Points, lines, and planes
    - Projection operation
    - Coordinates
    - Vector spaces
    - Vector space techniques
  - 5/ Linear transformations
    - Vector functions
    - Input and output spaces
    - Matrix representation of linear transformations
    - Column space and row spaces of matrix representations
    - Invertible matrix theorem
  - 6/ Theoretical linear algebra
    - Eigenvalues and eigenvectors
    - Special types of matrices
    - Abstract vectors paces
    - Abstract inner product spaces
    - Gram–Schmidt orthogonalization
    - Matrix decompositions
    - Linear algebra with complex numbers
  - 7/ Applications
  - 8/ Probability theory
  - 9/ Quantum mechanics
  - Notation appendix

```python
# helper code needed for running in colab
if 'google.colab' in str(get_ipython()):
    print('Downloading plot_helpers.py to util/ (only neded for colab')
    !mkdir util; wget https://raw.githubusercontent.com/minireference/noBSLAnotebooks/master/util/plot_helpers.py -P util
```

```python
# setup SymPy
from sympy import *
x, y, z, t = symbols('x y z t')
init_printing()

# a vector is a special type of matrix (an n-vector is either a nx1 or a 1xn matrix)
Vector = Matrix  # define alias Vector so I don't have to explain this during video
Point = Vector   # define alias Point for Vector since they're the same thing

# setup plotting
%matplotlib inline
import matplotlib.pyplot as mpl
from util.plot_helpers import plot_vec, plot_vecs, plot_line, plot_plane, autoscale_arrows
```

# 1/ Math fundamentals

Linear algebra builds upon high school math concepts like:
  - Numbers (integers, rationals, reals, complex numbers)
  - Functions ($f(x)$ takes an input $x$ and produces an output $y$)
  - Basic rules of algebra
  - Geometry (lines, curves, areas, triangles)
  - The cartesian plane

```python

```

# 2/ Intro to linear algebra

Linear algebra is the study of vectors and matrices.

## Vectors

```python
# define two vectors
u = Vector([2,3])
v = Vector([3,0])
u
```

```python
v
```

```python
plot_vecs(u, v)
autoscale_arrows()
```

## Vector operations

  - Addition (denoted $\vec{u}+\vec{v}$)
  - Subtraction, the inverse of addition (denoted $\vec{u}-\vec{v}$)
  - Scaling (denoted $\alpha \vec{u}$)
  - Dot product (denoted $\vec{u} \cdot \vec{v}$)
  - Cross product (denoted $\vec{u} \times \vec{v}$)

### Vector addition

```python
# algebraic
u+v
```

```python
# graphical
plot_vecs(u, v)
plot_vec(v, at=u, color='b')
plot_vec(u+v, color='r')
autoscale_arrows()
```

### Basis

When we describe the vector as the coordinate pair $(4,6)$, we're implicitly using the *standard basis* $B_s = \{ \hat{\imath}, \hat{\jmath} \}$. The vector $\hat{\imath} \equiv (1,0)$ is a unit-length vector in the $x$-direciton,
and $\hat{\jmath} \equiv (0,1)$ is a unit-length vector in the $y$-direction.

To be more precise when referring to vectors, we can indicate the basis as a subscript of every cooridnate vector $\vec{v}=(4,6)_{B_s}$, which tells $\vec{v}= 4\hat{\imath}+6\hat{\jmath}=4(1,0) +6(0,1)$.

```python
# the standard basis
ihat = Vector([1,0])
jhat = Vector([0,1])

v = 4*ihat + 6*jhat
v
```

```python
# geomtrically...
plot_vecs(ihat, jhat, 4*ihat, 6*jhat, v)
autoscale_arrows()
```

The same vector $\vec{v}$ will correspond to the a different pair of coefficients if a differebt basis is used.
For example, if we use the basis $B^\prime = \{ (1,1), (1,-1) \}$, the same vector $\vec{v}$ must be expressed as $\vec{v} = 5\vec{b}_1 +(-1)\vec{b}_2=(5,-1)_{B^\prime}$.

```python
# another basis B' = { (1,1), (1,-1) }
b1 = Vector([ 1, 1])
b2 = Vector([ 1, -1])

v = 5*b1 + (-1)*b2
v

# How did I know 5 and -1 are the coefficients w.r.t basis {b1,b2}?
# Matrix([[1,1],[1,-1]]).inv()*Vector([4,6])
```

```python
# geomtrically...
plot_vecs(b1, b2, 5*b1, -1*b2, v)
autoscale_arrows()
```

## Matrix operations

  - Addition (denoted $A+B$)
  - Subtraction, the inverse of addition (denoted $A-B$)
  - Scaling by a constant $\alpha$  (denoted $\alpha A$)
  - Matrix-vector product (denoted $A\vec{x}$, related to linear transformations)
  - Matrix product (denoted $AB$)
  - Matrix inverse (denoted $A^{-1}$)
  - Trace (denoted $\textrm{Tr}(A)$)
  - Determinant (denoted $\textrm{det}(A)$ or $|A|$)

In linear algebra we'll extend the notion of funttion $f:\mathbb{R}\to \mathbb{R}$, to functions that act on vectors called *linear transformations*.  We can understand the properties of linear transformations $T$ in analogy with ordinary functions:

\begin{align*}
    \textrm{function }
     f:\mathbb{R}\to \mathbb{R}
         & \ \Leftrightarrow \,
         \begin{array}{l}
    \textrm{linear transformation }
    T:\mathbb{R}^{n}\! \to \mathbb{R}^{m}
    \end{array} \\
    \textrm{input }  x\in \mathbb{R}
         & \ \Leftrightarrow \
        \textrm{input }  \vec{x} \in \mathbb{R}^n           \\
    \textrm{output } f(x) \in \mathbb{R}
         & \ \Leftrightarrow \
     \textrm{output }      T(\vec{x})\in \mathbb{R}^m \\
    g\circ\! f \: (x) =  g(f(x))
         & \ \Leftrightarrow \
    %      \textrm{matrix product }
    S(T(\vec{x})) \\
    \textrm{function inverse } f^{-1}
         & \ \Leftrightarrow \
          \textrm{inverse transformation } T^{-1} \\
    \textrm{zeros of } f
         & \ \Leftrightarrow \
          \textrm{kernel of } T \\
    \textrm{image of } f
         & \ \Leftrightarrow \
             \begin{array}{l}
             \textrm{image of } T
              \end{array}
\end{align*}

## Linear property

$$
  T(a\mathbf{x}_1 + b\mathbf{x}_2) = aT(\mathbf{x}_1) + bT(\mathbf{x}_2)
$$

## Matrix-vector product representation of linear transformations

Equivalence between linear transformstions $T$ and matrices $M_T$:

$$
  T : \mathbb{R}^n \to \mathbb{R}^m
  \qquad
  \Leftrightarrow
  \qquad
  M_T \in \mathbb{R}^{m \times n}
$$

$$
  \vec{y} = T(\vec{x})
  \qquad
  \Leftrightarrow
  \qquad
  \vec{y} = M_T\vec{x}
$$

# 3/ Computational linear algebra

## Gauss-Jordan elimination procedure

Suppose you're asked to solve for $x_1$ and $x_2$ in the following system of equations

\begin{align*}
1x_1 + 2x_2 &= 5 \\
3x_1 + 9x_2 &= 21.
\end{align*}

```python
# represent as an augmented matrix
AUG = Matrix([
  [1, 2,   5],
  [3, 9,  21]])
AUG
```

```python
# eliminate x_1 in second equation by subtracting 3x times the first equation
AUG[1,:] = AUG[1,:] - 3*AUG[0,:]
AUG
```

```python
# simplify second equation by dividing by 3
AUG[1,:] = AUG[1,:]/3
AUG
```

```python
# eliminate x_2 from first equation by subtracting 2x times the second equation
AUG[0,:] = AUG[0,:] - 2*AUG[1,:]
AUG
```

This augmented matrix is in *reduced row echelon form* (RREF), and corresponds to the system of equations:

\begin{align*}
1x_1  \ \  \qquad &= 1 \\
       1x_2 &= 2,
\end{align*}

so the the solution is $x_1=1$ and $x_2=2$.

## Matrix equations

See **page 177** in v2.2 of the book.

## Matrix product

```python
a,b,c,d,e,f, g,h,i,j = symbols('a b c d e f  g h i j')
A = Matrix([[a,b],
            [c,d],
            [e,f]])
B = Matrix([[g,h],
            [i,j]])

A, B
```

```python
A*B
```

```python
def mat_prod(A, B):
    """Compute the matrix product of matrices A and B."""
    assert A.cols == B.rows, "Error: matrix dimensions not compatible."
    m, ell = A.shape  # A is a  m x ell  matrix
    ell, n = B.shape  # B is a  ell x n  matrix
    C = zeros(m,n)
    for i in range(0,m):
        for j in range(0,n):
            C[i,j] = A[i,:].dot(B[:,j])
    return C

mat_prod(A,B)
```

```python
# mat_prod(B,A)
```

## Determinant

```python
a, b, c, d  = symbols('a b c d')
A = Matrix([[a,b],
            [c,d]])

A.det()
```

```python
# Consider the parallelogram with sides:
u1 = Vector([3,0])
u2 = Vector([2,2])

plot_vecs(u1,u2)
plot_vec(u1, at=u2, color='k')
plot_vec(u2, at=u1, color='b')
autoscale_arrows()

# What is the area of this parallelogram?
```

```python
# base = 3, height = 2, so area is 6
```

```python
# Compute the area of the parallelogram with sides u1 and u2 using the deteminant
A = Matrix([[3,0],
            [2,2]])
A.det()
```

## Matrix inverse

For an invertible matrix $A$, the matrix inverse $A^{-1}$ acts to undo the effects of $A$:

$$
 A^{-1} A \vec{v} = \vec{v}.
$$

The effect applying $A$ followed by $A^{-1}$ (or the other way around) is the identity transformation:

$$
 A^{-1}A \ = \ \mathbb{1} \ = \ AA^{-1}.
$$

```python
A = Matrix([[1, 2],
            [3, 9]])
A
```

```python
# Compute deteminant to check if inverse matrix exists
A.det()
```

The deteminant is non-zero so inverse exists.

```python
A.inv()
```

```python
A.inv()*A
```

### Adjugate-matrix formula

The *adjugate matrix* of the matrix $A$ is obtained by replacing each entry of the matrix with a partial determinant calculation (called *minors*). The minor $M_{ij}$ is the determinant of $A$ with its $i$th row and $j$th columns removed.

```python
A.adjugate() / A.det()
```

### Augmented matrix approach

$$
 \left[ \, A \, | \,  \mathbb{1} \, \right]
 \qquad
 -\textrm{Gauss-Jordan elimination}\rightarrow
 \qquad
 \left[ \, \mathbb{1} \, | \,  A^{-1} \, \right]
$$

```python
AUG = A.row_join(eye(2))
AUG
```

```python
# perform row operations until left side of AUG is in RREF
AUG[1,:] = AUG[1,:] - 3*AUG[0,:]
AUG[1,:] = AUG[1,:]/3
AUG[0,:] = AUG[0,:] - 2*AUG[1,:]
AUG
```

```python
# the inverse of A is in the right side of RREF(AUG)
AUG[:,2:5]  # == A-inverse
```

```python
# verify A times A-inverse gives the identity matrix...
A*AUG[:,2:5]
```

### Using elementary matrices

Each row operation $\mathcal{R}_i$ can be represented as an elementary matrix $E_i$. The elementary matrix of a given row operation is obtained by performing the row operation on the identity matrix.

```python
E1 = eye(2)
E1[1,:] = E1[1,:] - 3*E1[0,:]

E2 = eye(2)
E2[1,:] = E2[1,:]/3

E3 = eye(2)
E3[0,:] = E3[0,:] - 2*E3[1,:]

E1, E2, E3
```

```python
# the sequence of three row operations transforms the matrix A into RREF
E3*E2*E1*A
```

Recall definition $A^{-1}A=\mathbb{1}$, and we just observed that $E_3E_2E_1 A =\mathbb{1}$, so it must be that $A^{-1}=E_3E_2E_1$.

```python
E3*E2*E1
```

# 4/ Geometrical linear algebra

Points, lines, and planes are geometrical objects that are conveniently expressed using the language of vectors.

## Points

A point $p=(p_x,p_y,p_z)$ refers to a single location in $\mathbb{R}^3$.

```python
p = Point([2,4,5])
p
```

## Lines

A line is a one dimensional infinite subset of $\mathbb{R}^3$ that can be described as

$$
   \ell:  \{ p_o + \alpha \vec{v} \ | \ \forall \alpha \in \mathbb{R} \}.
$$

```python
po = Point([1,1,1])
v = Vector([1,1,0])

plot_line(v, po)
```

## Planes

A plane is a two-dimensional infinite subset of $\mathbb{R}^3$ that can be described in one of three ways:

The *general equation*:

$$
  P: \left\{ \,  Ax+By+Cz=D \, \right\}
$$

The *parametric equation*:

$$
  P: \{  p_{\textrm{o}}+s\,\vec{v} + t\,\vec{w}, \ \forall s,t \in \mathbb{R} \},
$$

which defines a plane that that contains the point $p_{\textrm{o}}$ and the vectors $\vec{v}$ and $\vec{w}$.

Or the *geometric equation*:

$$
  P: \left\{ \vec{n} \cdot [ (x,y,z) - p_{\textrm{o}} ] = 0 \,\right\},
$$

which defines a plane that contains point $p_{\textrm{o}}$ and has normal vector $\hat{n}$.

```python
# plot plane     2x + 1y + 1z = 5
normal = Vector([2,   1,   1])
D = 5

plot_plane(normal, D)
```

## Projection operation

A projection of the vector $\vec{v}$ in the direction $\vec{d}$ is denoted $\Pi_{\vec{d}}(\vec{v})$. The formula for computing the projections uses the dot product operation:

$$
   \Pi_{\vec{d}}(\vec{v})
      \ \equiv \
      (\vec{v} \cdot \hat{d}) \hat{d}
      \ =  \
      \left(\vec{v} \cdot \frac{\vec{d}}{\|\vec{d}\|} \right) \frac{\vec{d}}{\|\vec{d}\|}.
$$

```python
def proj(v, d):
    """Computes the projection of vector `v` onto direction `d`."""
    return v.dot( d/d.norm() )*( d/d.norm() )
```

```python
v = Vector([2,2])
d = Vector([3,0])
proj_v_on_d = proj(v,d)

plot_vecs(d, v, proj_v_on_d)
autoscale_arrows()
```

The basic projection operation can be used to compute projection onto planes, and compute distances between geomteirc objects (page 192).

## Bases and coordinate projections

See [page 225](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=68) in v2.2 of the book:

  - Different types of bases
    - Orthonormal
    - Orthogonal
    - Generic
  - Change of basis operation

## Vector spaces

See **page 231** in v2.2 of the book.

## Vector space techniques

See **page 244** in the book.

# 5/ Linear transformations

See [page 257](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=70) in v2.2 of the book.

## Vector functions

Functions that take vectors as inputs and produce vectors as outputs:

$$
  T:\mathbb{R}^{n}\! \to \mathbb{R}^{m}
$$

## Matrix representation of linear transformations

$$
  T : \mathbb{R}^n \to \mathbb{R}^m
  \qquad
  \Leftrightarrow
  \qquad
  M_T \in \mathbb{R}^{m \times n}
$$

## Input and output spaces

We can understand the properties of linear transformations $T$, and their matrix representations $M_T$ in analogy with ordinary functions:

\begin{align*}
    \textrm{function }
     f:\mathbb{R}\to \mathbb{R}
         & \ \Leftrightarrow \,
         \begin{array}{l}
    \textrm{linear transformation }
    T:\mathbb{R}^{n}\! \to \mathbb{R}^{m}		 \\
    \textrm{represented by the matrix } M_T \in \mathbb{R}^{m \times n}
    \end{array} \\
    %
    \textrm{input }  x\in \mathbb{R}
         & \ \Leftrightarrow \
        \textrm{input }  \vec{x} \in \mathbb{R}^n           \\
    %\textrm{compute }
    \textrm{output } f(x) \in \mathbb{R}
         & \ \Leftrightarrow \
    %      \textrm{compute matrix-vector product }
     \textrm{output }      T(\vec{x}) \equiv M_T\vec{x} \in \mathbb{R}^m \\
    %\textrm{function composition  }
    g\circ\! f \: (x) =  g(f(x))
         & \ \Leftrightarrow \
    %      \textrm{matrix product }
    S(T(\vec{x})) \equiv M_SM_T \vec{x} \\
    \textrm{function inverse } f^{-1}
         & \ \Leftrightarrow \
          \textrm{matrix inverse } M_T^{-1} \\
    \textrm{zeros of } f
         & \ \Leftrightarrow \
          \textrm{kernel of } T  \equiv  \textrm{null space of } M_T \equiv \mathcal{N}(A)  \\
    \textrm{image of } f
         & \ \Leftrightarrow \
             \begin{array}{l}
             \textrm{image of } T \equiv  \textrm{column space of } M_T  \equiv \mathcal{C}(A)
              \end{array}
\end{align*}

Observe we refer to the linear transformation $T$ and its matrix representation $M_T$ interchangeably.

## Finding matrix representations

See [page 269](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=74) in v2.2 of the book.

## Invertible matrix theorem

See [page 288](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=78) in the book.

# 6/ Theoretical linear algebra

## Eigenvalues and eigenvectors

An eigenvector of the matirx $A$ is a special input vector, for which the matrix $A$ acts as a scaling:

$$
  A\vec{e}_\lambda = \lambda\vec{e}_\lambda,
$$

where $\lambda$ is called the *eigenvalue* and $\vec{e}_\lambda$ is the corresponding eigenvector.

```python
A = Matrix([[1, 5],
            [5, 1]])
A
```

```python
A*Vector([1,0])
```

```python
A*Vector([1,1])
```

The *characterisitic polynomial* of the matrix $A$ is defined as

$$
  p(\lambda) \equiv \det(A-\lambda \mathbb{1}).
$$

```python
l = symbols('lambda')
(A-l*eye(2)).det()
```

```python
# the roots of the characteristic polynomial are the eigenvalues of A
solve( (A-l*eye(2)).det(), l)
```

```python
# or call `eigenvals` method
A.eigenvals()
```

```python
A.eigenvects()
# can also find eigenvects using (A-6*eye(2)).nullspace() and (A+4*eye(2)).nullspace()
```

```python
Q, Lambda = A.diagonalize()
Q, Lambda
```

```python
Q*Lambda*Q.inv()  # == eigendecomposition of A
```

## Special types of matrices

See [page 312](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=83) in v2.2 of the book.

## Abstract vectors paces

Generalize vector techniques to other vector like quantities. Allow us to talk about basis, dimention, etc.

See [page 318](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=84) in the book.

## Abstract inner product spaces

Use geometrical notions like length and orthogonaloty for abstract vectors.

See **page 322** in the book.

## Gram–Schmidt orthogonalization

See **page 328**.

## Matrix decompositions

See **page 332**.

## Linear algebra with complex numbers

See [page 339](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=88) in v2.2 of the book.

# Applications chapters

  - Chapter 7: Applications
  - Chapter 8: Probability theory
  - Chapter 9: Quantum mechanics

# Notation appendix

Check out [page 571](https://minireference.com/static/excerpts/noBSLA_v2_preview.pdf#page=142) in the book.