# Programación lineal

> La programación lineal es el campo de la optimización matemática dedicado a maximizar o minimizar (optimizar) funciones lineales, denominada función objetivo, de tal forma que las variables de dicha función estén sujetas a una serie de restricciones expresadas mediante un sistema de ecuaciones o inecuaciones también lineales.

**Referencias:**
- https://es.wikipedia.org/wiki/Programaci%C3%B3n_lineal
- https://docs.scipy.org/doc/scipy/reference/

## 1. Apuntes históricos

- 1826:	Joseph Fourier anticipa la programación lineal. Carl Friedrich Gauss resuelve ecuaciones lineales por eliminación "gaussiana".
- 1902:	Gyula Farkas concibe un método para resolver sistemas de inecuaciones.
- Es hasta la Segunda Guerra Mundial que se plantea la programación lineal como un modelo matemático para planificar gastos y retornos, de modo que se reduzcan costos de guerra y aumentar pérdidas del enemigo. Secreto hasta 1947 (posguerra).
- 1947:	George Dantzig publica el algoritmo simplex y John von Neumann desarrolló la teoría de la dualidad. Se sabe que Leonid Kantoróvich también formuló la teoría en forma independiente.
- Fue usado por muchas industrias en la planificación diaria.

**Hasta acá, tiempos exponenciales de solución. Lo siguiente, tiempo polinomial.**

- 1979: Leonid Khachiyan, diseñó el llamado Algoritmo del elipsoide, a través del cual demostró que el problema de la programación lineal es resoluble de manera eficiente, es decir, en tiempo polinomial.
- 1984: Narendra Karmarkar introduce el método del punto interior para resolver problemas de programación lineal.

**Mencionar complejidad computacional.**

## 2. Motivación

Ya la clase pasada habíamos mencionado que cuando se quería optimizar una función de varias variables con restricciones, se podía aplicar siempre el método de Multiplicadores de Lagrange. Sin embargo, este método es computacionalmente muy complejo conforme crece el número de variables.

Por tanto, cuando la función a optimizar y las restricciones son de caracter lineal, los métodos de solución que se pueden desarrollar son computacionalmente eficientes, por lo que es útil realizar la distinción.

## 3. Problemas de programación lineal

### 3.1. Ejemplo básico
Una compañía produce dos productos ($X_1$ y $X_2$) usando dos máquinas ($A$ y $B$). Cada unidad de $X_1$ que se produce requiere 50 minutos en la máquina $A$ y 30 minutos en la máquina $B$. Cada unidad de $X_2$ que se produce requiere 24 minutos en la máquina $A$ y 33 minutos en la máquina $B$.

Al comienzo de la semana hay 30 unidades de $X_1$ y 90 unidades de $X_2$ en inventario. El tiempo de uso disponible de la máquina $A$ es de 40 horas y el de la máquina $B$ es de 35 horas.

La demanda para $X_1$ en la semana actual es de 75 unidades y de $X_2$ es de 95 unidades. La política de la compañía es maximizar la suma combinada de unidades de $X_1$ e $X_2$ en inventario al finalizar la semana.

Formular el problema de decidir cuánto hacer de cada producto en la semana como un problema de programación lineal.

#### Solución

Sean:
- $x_1$ la cantidad de unidades de $X_1$ a ser producidas en la semana, y
- $x_2$ la cantidad de unidades de $X_2$ a ser producidas en la semana.

Notar que lo que se quiere es maximizar $x_1+x_2$.

Restricciones:
1. El tiempo de uso disponible de la máquina $A$ es de 40 horas: $50x_1+24x_2\leq 40(60)\Rightarrow 50x_1+24x_2\leq 2400$.
2. El tiempo de uso disponible de la máquina $B$ es de 35 horas: $30x_1+33x_2\leq 35(60)\Rightarrow 30x_1+33x_2\leq 2100$.
3. La demanda para $X_1$ en la semana actual es de 75 unidades: $x_1+30\geq 75\Rightarrow x_1\geq 45\Rightarrow -x_1\leq -45$.
4. La demanda para $X_2$ en la semana actual es de 95 unidades: $x_2+90\geq 95\Rightarrow x_2\geq 5\Rightarrow -x_2\leq -5$.

Finalmente, el problema puede ser expresado en la forma explicada como:
\begin{equation}
\begin{array}{ll}
\min_{x_1,x_2} & -x_1-x_2 \\
\text{s. a. }  & 50x_1+24x_2\leq 2400 \\
               & 30x_1+33x_2\leq 2100 \\
               & -x_1\leq -45 \\
               & -x_2\leq -5,
\end{array}
\end{equation}

o, eqivalentemente
\begin{equation}
\begin{array}{ll}
\min_{\boldsymbol{x}} & \boldsymbol{c}^T\boldsymbol{x} \\
\text{s. a. }        & \boldsymbol{A}_{eq}\boldsymbol{x}=\boldsymbol{b}_{eq} \\
                     & \boldsymbol{A}\boldsymbol{x}\leq\boldsymbol{b},
\end{array}
\end{equation}
con
- $\boldsymbol{c}=\left[-1 \quad -1\right]^T$,
- $\boldsymbol{A}=\left[\begin{array}{cc}50 & 24 \\ 30 & 33\\ -1 & 0\\ 0 & -1\end{array}\right]$, y
- $\boldsymbol{b}=\left[2400\quad 2100\quad -45\quad -5\right]^T$.

Preferiremos, en adelante, la notación vectorial/matricial.

### 3.2. En general
De acuerdo a lo descrito anteriormente, un problema de programación lineal puede escribirse en la siguiente forma:

\begin{equation}
\begin{array}{ll}
\min_{x_1,\dots,x_n} & c_1x_1+\dots+c_nx_n \\
\text{s. a. }        & a^{eq}_{j,1}x_1+\dots+a^{eq}_{j,n}x_n=b^{eq}_j \text{ para } 1\leq j\leq m_1 \\
                     & a_{k,1}x_1+\dots+a_{k,n}x_n\leq b_k \text{ para } 1\leq k\leq m_2,
\end{array}
\end{equation}
donde:
- $x_i$ para $i=1,\dots,n$ son las incógnitas o variables de decisión,
- $c_i$ para $i=1,\dots,n$ son los coeficientes de la función a optimizar,
- $a^{eq}_{j,i}$ para $j=1,\dots,m_1$ e $i=1,\dots,n$, son los coeficientes de la restricción de igualdad,
- $a_{k,i}$ para $k=1,\dots,m_2$ e $i=1,\dots,n$, son los coeficientes de la restricción de desigualdad,
- $b^{eq}_j$ para $j=1,\dots,m_1$ son valores conocidos que deben ser respetados estrictamente, y
- $b_k$ para $k=1,\dots,m_2$ son valores conocidos que no deben ser superados.

Equivalentemente, el problema puede escribirse como

\begin{equation}
\begin{array}{ll}
\min_{\boldsymbol{x}} & \boldsymbol{c}^T\boldsymbol{x} \\
\text{s. a. }        & \boldsymbol{A}_{eq}\boldsymbol{x}=\boldsymbol{b}_{eq} \\
                     & \boldsymbol{A}\boldsymbol{x}\leq\boldsymbol{b},
\end{array}
\end{equation}
donde:
- $\boldsymbol{x}=\left[x_1\quad\dots\quad x_n\right]^T$,
- $\boldsymbol{c}=\left[c_1\quad\dots\quad c_n\right]^T$,
- $\boldsymbol{A}_{eq}=\left[\begin{array}{ccc}a^{eq}_{1,1} & \dots & a^{eq}_{1,n}\\ \vdots & \ddots & \vdots\\ a^{eq}_{m_1,1} & \dots & a^{eq}_{m_1,n}\end{array}\right]$,
- $\boldsymbol{A}=\left[\begin{array}{ccc}a_{1,1} & \dots & a_{1,n}\\ \vdots & \ddots & \vdots\\ a_{m_2,1} & \dots & a_{m_2,n}\end{array}\right]$,
- $\boldsymbol{b}_{eq}=\left[b^{eq}_1\quad\dots\quad b^{eq}_{m_1}\right]^T$, y
- $\boldsymbol{b}=\left[b_1\quad\dots\quad b_{m_2}\right]^T$.

**Nota:** el problema $\max_{\boldsymbol{x}}\boldsymbol{g}(\boldsymbol{x})$ es equivalente a $\min_{\boldsymbol{x}}-\boldsymbol{g}(\boldsymbol{x})$.

#### Bueno, y una vez planteado, ¿cómo se resuelve el problema?

Este problema está sencillo pues solo es en dos variables. La solución gráfica es válida.

```python
# Importar el módulo pyplot de la librería matplotlib, y la librería numpy

```

```python
# Definir las funciones frontera de la región factible

```

```python
# Vector independiente x1 y funciones evaluadas

```

```python
# Gráfica de la región factible y esquinas

```

```python

```

De acuerdo a la solución gráfica, debemos producir:

- 45 unidades de $X_1$, y
- 6.25 unidades de $X_2$.

Además, el inventario total de productos $X_1$ y $X_2$ al final de la semana es de 1.25 unidades.

**Actividad.** Mónica hace aretes y cadenitas de joyería. Es tan buena, que todo lo que hace lo vende.

Le toma 30 minutos hacer un par de aretes y una hora hacer una cadenita, y como Mónica también es estudihambre, solo dispone de 10 horas a la semana para hacer las joyas. Por otra parte, el material que compra solo le alcanza para hacer 15 unidades (el par de aretes cuenta como unidad) de joyas por semana.

La utilidad que le deja la venta de las joyas es $\$15$ en cada par de aretes y $\$20$ en cada cadenita.

¿Cuántos pares de aretes y cuántas cadenitas debería hacer Mónica para maximizar su utilidad?

Formular el problema en la forma explicada y obtener la solución gráfica (puede ser a mano).

**Diez minutos: quien primero lo haga, y me lo mande bien explicado al correo, tiene 20 puntos para el examen final.**

## 5. ¿Cómo se resuelve en python?

### 5.1 Librería `SciPy`

`SciPy` es un software de código abierto basado en `Python` para matemáticas, ciencia e ingeniería.

En particular, los siguientes son algunos de los paquetes básicos:
- `NumPy`
- **Librería `SciPy`**
- `SymPy`
- `matplotlib`
- `pandas`

La **Librería `SciPy`** es uno de los paquetes principales y provee varias rutinas numéricas eficientes. Entre ellas, para integración numérica y optimización.

En esta clase, y en lo que resta del módulo, estaremos utilizando el módulo `optimize` de la librería `SciPy`.

**Importémoslo**

```python
# Importar el módulo optimize de la librería scipy

```

El módulo `optimize` que acabamos de importar contiene varias funciones para optimización y búsqueda de raices ($f(x)=0$). Entre ellas se encuentra la función `linprog`

```python
# Función linprog del módulo optimize

```

\begin{align}
\min_x \ & c^T x \\
\mbox{such that} \ & A_{ub} x \leq b_{ub},\\
        & A_{eq} x = b_{eq},\\
        & l \leq x \leq u ,
\end{align}

la cual resuelve problemas como los que aprendimos a plantear.

### 5.2 Solución del ejemplo básico con linprog
Ya hicimos la solución gráfica. Contrastemos con la solución que nos da `linprog`...

- $\boldsymbol{c}=\left[-1 \quad -1\right]^T$,
- $\boldsymbol{A}=\left[\begin{array}{cc}50 & 24 \\ 30 & 33\\ -1 & 0\\ 0 & -1\end{array}\right]$, y
- $\boldsymbol{b}=\left[2400\quad 2100\quad -45\quad -5\right]^T$.

```python
# Importar numpy para crear las matrices

```

```python
# Crear las matrices para resolver el problema

```

```python

```

```python
# Resolver utilizando linprog

```

```python
# Mostrar el resultado

```

```python
# Extraer el vector solución

```

**Conclusión**
- 45 unidades de $X_1$, y
- 6.25 unidades de $X_2$.

Además, el inventario total de productos $X_1$ y $X_2$ al final de la semana es de 1.25 unidades.

**Otra forma:** poner las cotas de las variables a parte

```python
# Escribir matrices y cotas

# Cotas

```

```python
# Resolver

```

```python
# Mostrar el resultado

```

**Actividad.** Resolver el ejemplo de Mónica y sus tiliches con `linprog`

```python
# Resolver acá

```

```python

```

```python

```

## 6. Problema de transporte 1
- **Referencia**: https://es.wikipedia.org/wiki/Programaci%C3%B3n_lineal

Este es un caso curioso, con solo 6 variables (un caso real de problema de transporte puede tener fácilmente más de 1.000 variables) en el cual se aprecia la utilidad de este procedimiento de cálculo.

Existen tres minas de carbón cuya producción diaria es:
- la mina "a" produce 40 toneladas de carbón por día;
- la mina "b" produce 40 t/día; y,
- la mina "c" produce 20 t/día.

En la zona hay dos centrales termoeléctricas que consumen:
- la central "d" consume 40 t/día de carbón; y,
- la central "e" consume 60 t/día.

Los costos de mercado, de transporte por tonelada son:
- de "a" a "d" = 2 monedas;
- de "a" a "e" = 11 monedas;
- de "b" a "d" = 12 monedas;
- de "b" a "e" = 24 monedas;
- de "c" a "d" = 13 monedas; y,
- de "c" a "e" = 18 monedas.

Si se preguntase a los pobladores de la zona cómo organizar el transporte, tal vez la mayoría opinaría que debe aprovecharse el precio ofrecido por el transportista que va de "a" a "d", porque es más conveniente que los otros, debido a que es el de más bajo precio.

En este caso, el costo total del transporte es:
- transporte de 40 t de "a" a "d" = 80 monedas;
- transporte de 20 t de "c" a "e" = 360 monedas; y,
- transporte de 40 t de "b" a "e" = 960 monedas,

Para un total 1.400 monedas.

Sin embargo, formulando el problema para ser resuelto por la programación lineal con
- $x_1$ toneladas transportadas de la mina "a" a la central "d"
- $x_2$ toneladas transportadas de la mina "a" a la central "e"
- $x_3$ toneladas transportadas de la mina "b" a la central "d"
- $x_4$ toneladas transportadas de la mina "b" a la central "e"
- $x_5$ toneladas transportadas de la mina "c" a la central "d"
- $x_6$ toneladas transportadas de la mina "c" a la central "e"

se tienen las siguientes ecuaciones:

Restricciones de la producción:

- $x_1 + x_2 \leq 40$
- $x_3 + x_4 \leq 40$
- $x_5 + x_6 \leq 20$

Restricciones del consumo:

- $x_1 + x_3 + x_5 \geq 40$
- $x_2 + x_4 + x_6 \geq 60$

La función objetivo será:

$$\min_{x_1,\dots,x_6}2x_1 + 11x_2 + 12x_3 + 24x_4 + 13x_5 + 18x_6$$

Resolver con `linprog`

```python
# Matrices y cotas

# bnds = ((0, None), (0, None), (0, None), (0, None), (0, None), (0, None))

```

```python
# Resolver

```

```python

```

```python

```

```python
# Mostrar resultado

```

```python

```

```python

```

**Conclusión**

- Se deben transportar 40 ton. de la mina "a" a la central "e" ($x_2$)
- Se deben transportar 40 ton. de la mina "b" a la central "d" ($x_3$)
- Se deben transportar 20 ton. de la mina "c" a la central "e" ($x_6$)

## 7. Optimización de inversión en bonos

**Referencia:**

```python
from IPython.display import YouTubeVideo
YouTubeVideo('gukxBus8lOs')
```

El objetivo de este problema es determinar la mejor estrategia de inversión, dados diferentes tipos de bono, la máxima cantidad que puede ser invertida en cada bono, el porcentaje de retorno y los años de madurez. También hay una cantidad fija de dinero disponible ($\$750,000$). Por lo menos la mitad de este dinero debe ser invertido en bonos con 10 años o más para la madurez. Se puede invertir un máximo del $25\%$ de esta cantidad en cada bono. Finalmente, hay otra restricción que no permite usar más de $35\%$ en bonos de alto riesgo.

Existen seis (6) opciones de inversión con las letras correspondientes $A_i$

1. $A_1$:(Tasa de retorno=$8.65\%$; Años para la madurez=11, Riesgo=Bajo)
1. $A_2$:(Tasa de retorno=$9.50\%$; Años para la madurez=10, Riesgo=Alto)
1. $A_3$:(Tasa de retorno=$10.00\%$; Años para la madurez=6, Riesgo=Alto)
1. $A_4$:(Tasa de retorno=$8.75\%$; Años para la madurez=10, Riesgo=Bajo)
1. $A_5$:(Tasa de retorno=$9.25\%$; Años para la madurez=7, Riesgo=Alto)
1. $A_6$:(Tasa de retorno=$9.00\%$; Años para la madurez=13, Riesgo=Bajo)

Lo que se quiere entonces es maximizar el retorno que deja la inversión.

Este problema puede ser resuelto con programación lineal. Formalmente, puede ser descrito como:

$$\max_{A_1,A_2,...,A_6}\sum^{6}_{i=1} A_iR_i,$$

donde $A_i$ representa la cantidad invertida en la opción, y $R_i$ representa la tasa de retorno respectiva.

Plantear restricciones...

```python
# Matrices y cotas

```

```python
# Resolver

```

```python
# Mostrar resultado

```

Recordar que en el problema minimizamos $-\sum^{6}_{i=1} A_iR_i$. El rendimiento obtenido es entonces:

```python

```

**Conclusión**

-

## 8. Tarea

### 1. Diseño de la Dieta Óptima

Se quiere producir comida para gatos de la manera más barata, no obstante se debe también asegurar que se cumplan los datos requeridos de analisis nutricional. Por lo que se quiere variar la cantidad de cada ingrediente para cumplir con los estandares nutricionales. Los requisitos que se tienen es que en 100 gramos, se deben tener por lo menos 8 gramos de proteína y 6 gramos de grasa. Así mismo, no se debe tener más de 2 gramos de fibra y 0.4 gramos de sal.

Los datos nutricionales se pueden obtener de la siguiente tabla:

|Ingrediente|Proteína|Grasa|Fibra|Sal|
|----|---- |---- |---- |---- |
|Pollo|  10.0%|08.0%|00.1%|00.2%
|Carne|  20.0%|10.0%|00.5%|00.5%
|Cordero|15.0%|11.0%|00.5%|00.7%
|Arroz|  00.0%|01.0%|10.0%|00.2%
|Trigo|  04.0%|01.0%|15.0%|00.8%
|Gel|    00.0%|00.0%|00.0%|00.0%

Los costos de cada producto son:

Ingrediente|Costo por gramo
:----|----
Pollo|$\$$0.013
Carne|$\$$0.008
Cordero|$\$$0.010
Arroz|$\$$0.002
Trigo|$\$$0.005
Gel|$\$$0.001

Lo que se busca optimizar en este caso es la cantidad de productos que se debe utilizar en la comida de gato, para simplificar la notación se van a nombrar las siguientes variables:

$x_1:$ Gramos de pollo
$x_2:$ Gramos de carne
$x_3:$ Gramos de cordero
$x_4:$ Gramos de arroz
$x_5:$ Gramos de trigo
$x_6:$ Gramos de gel

Con los datos, se puede plantear la función objetivo, está dada por la siguiente expresión:

$$\min 0.013 x_1 + 0.008 x_2 + 0.010 x_3 + 0.002 x_4 + 0.005 x_5 + 0.001 x_6$$

Las restricciones estarían dadas por el siguiente conjunto de ecuaciones:

$x_1+x_2+x_3+x_4+x_5+x_6=100$

$(10.0 x_1+ 20.0 x_2+ 15.0 x_3+ 00.0 x_4+ 04.0 x_5+ 00.0 x_6)/100 \geq 8.0$

$(08.0 x_1+ 10.0 x_2+ 11.0 x_3+ 01.0 x_4+ 01.0 x_5+ 00.0 x_6)/100 \geq 6.0$

$(00.1 x_1+ 00.5 x_2+ 00.5 x_3+ 10.0 x_4+ 15.0 x_5+ 00.0 x_6)/100 \leq 2.0$

$(00.2 x_1+ 00.5 x_2+ 00.7 x_3+ 00.2 x_4+ 00.8 x_5+ 00.0 x_6)/100 \leq 0.4$

La primer condición asegura que la cantidad de productos que se usará cumple con los 100 gramos. Las siguientes sólo siguen los lineamientos planteados para cumplir con los requisitos nutrimentales.

### 2. Otro problema de transporte

Referencia: https://relopezbriega.github.io/blog/2017/01/18/problemas-de-optimizacion-con-python/

Supongamos que tenemos que enviar cajas de cervezas de 2 cervecerías (Modelo y Cuauhtémoc Moctezuma) a 5 bares de acuerdo al siguiente gráfico:

Asimismo, supongamos que nuestro gerente financiero nos informa que el costo de transporte por caja de cada ruta se conforma de acuerdo a la siguiente tabla:

```python
import pandas as pd
info = pd.DataFrame({'Bar1': [2, 3], 'Bar2': [4, 1], 'Bar3': [5, 3], 'Bar4': [2, 2], 'Bar5': [1, 3]}, index = ['CerveceriaA', 'CerveceriaB'])
info
```

Y por último, las restricciones del problema, van a estar dadas por las capacidades de oferta y demanda de cada cervecería (en cajas de cerveza) y cada bar, las cuales se detallan en el gráfico de más arriba.

Sean:
- $x_i$ cajas transportadas de la cervecería A al Bar $i$,
- $x_{i+5}$ cajas transportadas de la cervecería B al Bar $i$.

La tarea consiste en plantear el problema de minimizar el costo de transporte de la forma vista y resolverlo con `linprog`.

Deben crear un notebook de jupyter (archivo .ipynb) y llamarlo Tarea4_ApellidoNombre, y subirlo a moodle.

**Definir fecha**

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez.
</footer>


```python
# Symbols
from sympy import *
from sympy.plotting import plot_parametric
import numpy as np
import scipy
import scipy.special

init_printing(use_unicode=True)

x,y,z,t,mu,nu=symbols('x y z t mu nu')
k,m,n=symbols('k m n', integer=True)
f,g,h=symbols('f g h', cls=Function)

R=S.Reals
C=S.Complexes

i=sqrt(-1)
```

```python
from matplotlib import animation, rc
from IPython.display import HTML
import matplotlib.pyplot as plt
%config InlineBackend.figure_formats = ['svg']
%matplotlib inline
```

```python
plt.rcParams['figure.autolayout'] = False
plt.rcParams['figure.figsize'] = 11, 11
```

```python
from mpmath import *
from colorsys import hsv_to_rgb
```

## Visualizing the complex function

```python
def PerFract(x, t, m, M):
    x=x/t
    return m+(M-m)*(x-floor(x))

def color_clines(fz):# this is the color function that replaces the
                     #default  color function implemented in mpmath.cplot
    if isinf(fz):
        return (0, 0, 1.0)#hsv code
    if isnan(fz):
        return (0., 0., 0.5)

    n=15#n is the number of rays drawn in a cycle

    h = (float(arg(fz)) + pi) / (2*pi) #hue
    h = (h - 0.5) % 1.0

    Phc=PerFract(h, 1.0/n, 0.6, 1) #  set brightness between 0.6 and 1,
                                   #in order to avoid too dark colors
    modul=fabs(fz)
    Logm=log(modul)
    s = 0.9 #saturation

    Modc=PerFract(Logm, 2*pi/n, 0.6, 1)
    v=Modc*Phc

    return  hsv_to_rgb(h,s, v)
```

```python
cplot(zeta, re=[-15, 15], im=[-30, 30], color=color_clines, verbose=False, points=200000, file='riemann-zeta-[-15,15]-[-30,30].svg')
```

```python
cplot(zeta, re=[-3,3], im=[-3,3], color=color_clines, verbose=False, points=200000, file='riemann-zeta-[-3,3]-[-3,3].svg')
```

A view around the trivial zeroes $s = -2n$:

```python
cplot(zeta, re=[-8.5,0.5], im=[-0.5,0.5], color=color_clines, verbose=False, points=200000, file='riemann-zeta-[-8.5,0.5]-[-0.5,0.5].svg')
```

A view around the critical strip $\operatorname{Re}(s)\in\left(0,1\right)$:

```python
cplot(zeta, re=[-0.1,1.1], im=[-3,3], color=color_clines, verbose=False, points=200000, file='riemann-zeta-[-0.1,1.1]-[-3,3].svg')
```

```python
cplot(zeta, re=[-0.1,1.1], im=[-30,30], color=color_clines, verbose=False, points=200000, file='riemann-zeta-[-0.1,1.1]-[-30,30].svg')
```

Portions of this file are derived from [https://github.com/empet/Math/blob/d9a2187faaa06db53220ef6ff16f062f9ce4f3e1/Riemann-Zeta.ipynb](https://github.com/empet/Math/blob/d9a2187faaa06db53220ef6ff16f062f9ce4f3e1/Riemann-Zeta.ipynb).  See the LICENSE file in the root of this repository for details.

```python

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, C.D. Cooper, G.F. Forsyth.

# Spreading out

Welcome to the second lesson of the course module: "_Spreading out: parabolic PDEs."_ We're studying the heat equation in one spatial dimension:

\begin{equation}
\frac{\partial T}{\partial t} = \alpha \frac{\partial^2 T}{\partial x^2}
\end{equation}

where $\alpha$ is the thermal diffusivity and $T$ is the temperature.

In the previous lesson, we reviewed the numerical solution of the 1D diffusion equation with a forward-time, centered-space scheme: an _explicit_ scheme. What does that mean?

The solution for $T$ at timestep $t^{n+1}$ was calculated using different combinations of $T$ values from the *previous* timestep $t^n$.  We have complete knowledge of the parts that feed into the solution update at each spatial point.

*Implicit* methods work differently: we will use more data from the "future" in the update, including several values of $T$ at $t^{n+1}$. This will make the scheme more difficult to apply, but there are several reasons why it may be worth the effort.

In lesson 1, we discussed two disadvantages of explicit methods: (1) boundary effects drag behind by one time step; (2) stability requirements constrain the time-step to very small values. Both of these issues are resolved by implicit schemes.

## Implicit schemes

Let's move things around a bit and try combining the Euler time step with an evaluation of the spatial derivative on the *updated* solution at $t^{n+1}$. The discretized form of the equation is now as follows:

\begin{equation}\frac{T_{i}^{n+1}-T_{i}^{n}}{\Delta t}=\alpha\frac{T_{i+1}^{n+1}-2T_{i}^{n+1}+T_{i-1}^{n+1}}{\Delta x^2}\end{equation}

The stencil for this discretization doesn't look anything like the other stencils we've used until now. Check it out.

.
#### Figure 1. Stencil of the implicit central scheme.

From the previous time-step, we only know $T_i^{n}$, but what about $T_i^{n+1}$, $T_{i-1}^{n+1}$ and $T_{i+1}^{n+1}$?  What can we do?

No need to panic! Let's start by putting what we *do know* on the right-hand side of the equation and what we *don't know* on the left. We get:

\begin{equation} -T_{i-1}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{i}^{n+1} - T_{i+1}^{n+1} = T_{i}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

It looks like there are a lot of unknowns and just one equation!

What does it look like with $i=1$?

\begin{equation}-T_{0}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{1}^{n+1} - T_{2}^{n+1} = T_{1}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

and $i=2$?

\begin{equation}-T_{1}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{2}^{n+1} - T_{3}^{n+1} = T_{2}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

What about $i=3$?

\begin{equation}-T_{2}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{3}^{n+1} - T_{4}^{n+1} = T_{3}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

Can you see the common element across equations? Here's a little help:

$T_{i}^{n+1}$ also appears in the equation for $T_{i-1}^{n+1}$ and $T_{i+1}^{n+1}$. We might have enough equations if we apply this for all $i$-values at the same time, don't you think? In fact, this is a linear system of equations for the unknown values $T_{i}^{n+1}$ on the spatial grid.

### What about boundary conditions?

Let's look at the boundary points of the example from the previous lesson with a Dirichlet BC at $x=0$ and a Neumann BC at $x=1$, discretizing with $N$ mesh points.

The value $T_0^{n+1}$ is known at every time-step from the BC, so putting all unknown terms on the left-hand side of the equation and the known values on the right side yields the following for the $i=1$ equation:

\begin{equation}-T_{2}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{1}^{n+1} = T_{1}^{n}\frac{\Delta x^2}{\alpha\Delta t} + T_{0}^{n+1} \end{equation}

That was easy!

On the other hand, for $i=N-2$, the equation reads

\begin{equation} -T_{N-3}^{n+1} + \left( 2 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{N-2}^{n+1} - T_{N-1}^{n+1} = T_{N-2}^{n}\frac{\Delta x^2}{\alpha\Delta t} \end{equation}

The discretized Neumann boundary condition on the right side of the rod is

\begin{equation}\frac{T^{n}_{N-1} - T^{n}_{N-2}}{\Delta x} = q.\end{equation}

But we can just as easily write that at time step $n+1$ (the boundary conditions apply at every time-step):

\begin{equation}\frac{T^{n+1}_{N-1} - T^{n+1}_{N-2}}{\Delta x} = q.\end{equation}

Inserting the Neumann boundary condition in the equation for $i=N-2$ yields

\begin{equation} - T_{N-3}^{n+1} + \left( 1 + \frac{\Delta x^2}{\alpha\Delta t}\right) T_{N-2}^{n+1} = T_{N-2}^{n}\frac{\Delta x^2}{\alpha\Delta t} + \Delta x q \end{equation}

Make sure you work this out with pen and paper: it's important to recognize where these terms come from!

Now we can write the linear system of equations in matrix form as follows:

$$[A][x] = [b]+[b]_{b.c.}$$

where the matrix of coefficients $[A]$ is a sparse matrix—most of the matrix elements are zero—with three non-zero diagonals. We write below the system expanded out, so you can see the structure of the matrix, with $\sigma=\frac{\alpha\Delta t}{\Delta x^2}$:

\begin{align}\left[ \begin{array}{cccccc}
 \left(2 + \frac{1}{\sigma}\right) & -1 & 0 & \cdots & & 0 \\
 -1 & \left(2 + \frac{1}{\sigma}\right) & -1 & 0 & \cdots & 0 \\
 0 & & \ddots& & & \vdots \\
 \vdots & & & & \left(2 + \frac{1}{\sigma}\right)& \\
 0 & \cdots & & & -1 & \left(1 + \frac{1}{\sigma}\right) \end{array} \right]
 \cdot
 \left[ \begin{array}{c}
 T_1^{n+1} \\ T_2^{n+1} \\ \vdots \\ \\ T_{N-2}^{n+1} \end{array} \right]
 =
 \left[ \begin{array}{c}
T_1^n \frac{1}{\sigma} \\ T_2^{n}\frac{1}{\sigma} \\ \vdots \\ \\ T_{N-2}^{n}\frac{1}{\sigma} \end{array} \right]
 +
 \begin{bmatrix}
  T_0^{n+1}\\
  0\\\\
  \vdots\\\\
  0\\
  q\Delta x
 \end{bmatrix}
 \end{align}

Notice that the Dirichlet boundary condition adds only a term to the right-hand side of the system.  The Neumann boundary condition both adds a term to the right-hand side and modifies the matrix $[A]$.

### Problem set up

We'll re-use the problem from lesson 1: we have a graphite rod, with [thermal diffusivity](http://en.wikipedia.org/wiki/Thermal_diffusivity) $\alpha=1.22\times10^{-3} {\rm m}^2/{\rm s}$, length $L=1{\rm m}$, and temperature held at $T=100{\rm C}$ on the left end, $x=0$, and $0{\rm C}$ everywhere else intially. We'll compute the evolution of temperature on the length of the rod.

Let's start like we did in the previous lesson: import your libraries and set up the discretization.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

```python
L = 1.
nt = 100
nx = 51.
alpha = 1.22e-3

q = 0.

dx = L/(nx-1)

qdx = q*dx

Ti = numpy.zeros(nx)
Ti[0] = 100
```

    /home/gil/anaconda/lib/python3.5/site-packages/ipykernel/__main__.py:12: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future

### Solving a linear system

We need to solve the linear system of equations written above to advance the solution in time. Luckily, we can rely on our friends from SciPy who have developed some nice linear solvers, so we don't need to write our own.

From `scipy.linalg`, let's import `solve`: a function to solve linear systems. Make sure to explore the documentation of [`scipy.linalg`](http://docs.scipy.org/doc/scipy-0.14.0/reference/linalg.html). We'll need to define our own custom functions to generate the coefficient matrix and the right-hand side of the linear system. You should carefully study the code below.

```python
from scipy.linalg import solve
```

```python
def generateMatrix(N, sigma):
    """ Computes the matrix for the diffusion equation with backward Euler
        Dirichlet condition at i=0, Neumann at i=-1

    Parameters:
    ----------
    T: array of float
        Temperature at current time step
    sigma: float
        alpha*dt/dx^2

    Returns:
    -------
    A: 2D numpy array of float
        Matrix for diffusion equation
    """

    # Setup the diagonal
    d = numpy.diag(numpy.ones(N-2)*(2+1./sigma))

    # Consider Neumann BC
    d[-1,-1] = 1+1./sigma

    # Setup upper diagonal
    ud = numpy.diag(numpy.ones(N-3)*-1, 1)

    # Setup lower diagonal
    ld = numpy.diag(numpy.ones(N-3)*-1, -1)

    A = d + ud + ld

    return A
```

```python
def generateRHS(T, sigma, qdx):
    """ Computes right-hand side of linear system for diffusion equation
        with backward Euler

    Parameters:
    ----------
    T: array of float
        Temperature at current time step
    sigma: float
        alpha*dt/dx^2
    qdx: float
        flux at right boundary * dx

    Returns:
    -------
    b: array of float
        Right-hand side of diffusion equation with backward Euler
    """
    b = numpy.zeros_like(T)

    b = T[1:-1]*1./sigma
    # Consider Dirichlet BC
    b[0] += T[0]
    # Consider Neumann BC
    b[-2] += qdx

    return b
```

Next, we'll define a function that steps in time using the implicit central-space scheme.  Remember that for an implicit method, a step in time is performed by solving the entire linear system. This is a fundamental difference between implicit and explicit methods, and implies a considerable computational cost.

```python
def implicit_ftcs(T, A, nt, sigma, qdx):
    """ Advances diffusion equation in time with implicit central scheme

    Parameters:
    ----------
    T: array of float
        initial temperature profile
    A: 2D array of float
        Matrix with discretized diffusion equation
    nt: int
        number of time steps
    sigma: float
        alpha*td/dx^2

    qdx: float
        flux at right boundary * dx
    Returns:
    -------
    T: array of floats
        temperature profile after nt time steps
    """

    for t in range(nt):
        Tn = T.copy()
        b = generateRHS(Tn, sigma, qdx)
        # Use numpy.linalg.solve
        T_interior = solve(A,b)
        T[1:-1] = T_interior
        # Enforce Neumann BC (Dirichlet is enforced automatically)
        T[-1] = T[-2] + qdx

    return T
```

We solve the linear system for every time step, but the $A$ matrix does not change. Thus, you can generate it only once and then use it as an input parameter to the time-stepping function. Let's try this out!

```python
sigma = 0.5
dt = sigma * dx*dx/alpha
nt = 1000

A = generateMatrix(nx, sigma)
```

It's always a good idea to check your matrix...

```python
print(A)
```

    [[ 4. -1.  0. ...,  0.  0.  0.]
     [-1.  4. -1. ...,  0.  0.  0.]
     [ 0. -1.  4. ...,  0.  0.  0.]
     ...,
     [ 0.  0.  0. ...,  4. -1.  0.]
     [ 0.  0.  0. ..., -1.  4. -1.]
     [ 0.  0.  0. ...,  0. -1.  3.]]

Looks good!

Now, we apply the time iteration with our very own function `implicit_ftcs()`.

```python
T = implicit_ftcs(Ti.copy(), A, nt, sigma, qdx)
```

Now plot the solution!

```python
pyplot.plot(numpy.linspace(0,1,nx), T, color='#003366', ls='-', lw=3);
```

Not too impressive, this looks just like the result from *explicit* forward in time, centered in space for $\alpha\frac{\Delta t}{\Delta x^2} = \frac{1}{2}$.

But try $\alpha\frac{\Delta t}{\Delta x^2} = 5$, which violates the stability condition of the *explicit* scheme:

```python
sigma = 5.0

A = generateMatrix(nx, sigma)

T = implicit_ftcs(T.copy(), A, nt, sigma, qdx)

pyplot.plot(numpy.linspace(0,1,nx), T, color='#003366', ls='-', lw=3);
```

**It didn't blow up!**

We were not able to use such a large time step with the explicit scheme. You can try out other values of `sigma` and you'll get a stable solution. In fact, this is an *unconditionally stable* scheme—the most valuable feature of implicit methods is that they give stable solutions without a constraint on the choice of time step.

Using the implicit scheme, we can always advance in time using larger time steps. But each time step requires the solution of a linear system, which is computationally expensive. This is the trade-off between explicit and implicit methods.
To experiment further, set different values of the Neumann boundary flux and see if the solution behaves as you expect.

##### A word of warning

Implicit methods allow you to use significantly larger time steps, because they are not subject to stability constraints. But that doesn't mean you can use just _any_ large time step! Remember that Euler's method is a first-order method, so the _accuracy_ gets worse as you increase the time step, in direct proportion. In fact, you can lose the ability to capture the correct physics if your time step is too large. Numerical stability does not imply accuracy!

##### Dig deeper

You see how matrix `A` is mostly full of zeros?  We call such a matrix *sparse*, and there are many ways to make more efficient calculations taking advantage of their particular structure. First of all, you can optimize the memory usage. Check out SciPy's [sparse-matrix storage formats](http://docs.scipy.org/doc/scipy-0.14.0/reference/sparse.html): you don't need too store $(N-2)^2$ elements! For example, a `coo_matrix` format stores only $3*N_\text{nonzero}$, where $N_\text{nonzero}$ is the number of non-zero elements in `A`. Make sure to explore this topic a little more. It's an important topic in numerical PDEs.

---
###### The cell below loads the style of the notebook

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "PT Mono";
    font-size: 90%;
}

</style>


# US Production Data for RBC Modeling

```python
import pandas as pd
import numpy as np
import fredpy as fp
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
pd.plotting.register_matplotlib_converters()
```

```python
# Load API key
fp.api_key = fp.load_api_key('fred_api_key.txt')

# Download nominal GDP, nominal personal consumption expenditures, nominal
# gross private domestic investment, the GDP deflator, and an index of hours
# worked in the nonfarm business sector produced by the BLS. All data are
# from FRED and are quarterly.
gdp = fp.series('GDP')
cons = fp.series('PCEC')
invest = fp.series('GPDI')
hours = fp.series('HOANBS')
defl = fp.series('GDPDEF')
pcec = fp.series('PCEC')
m2 = fp.series('M2SL')
tb3mo = fp.series('TB3MS')
unemp = fp.series('UNRATE')

# Convert monthly M2, 3-mo T-Bill, and unemployment to quarterly
m2 = m2.as_frequency('Q')
tb3mo = tb3mo.as_frequency('Q')
unemp = unemp.as_frequency('Q')

# Convert unemployment and t-bill data to decimals instead of percents
unemp.data = unemp.data/100
tb3mo.data = tb3mo.data/100

# pcec inflation as pecent change over past year
pcec = pcec.apc()
pcec.data = pcec.data/100

# Make sure that all of the downloaded series have the same data ranges
gdp,cons,invest,hours,defl,pcec,m2,tb3mo,unemp = fp.window_equalize([gdp,cons,invest,hours,defl,pcec,m2,tb3mo,unemp])

# Compute real GDP, real consumption, real investment
gdp.data = gdp.data/defl.data*100
cons.data = cons.data/defl.data*100
invest.data = invest.data/defl.data*100
m2.data = m2.data/defl.data*100

# Print units
print('Hours units:   ',hours.units)
print('Deflator units:',defl.units)
```

    Hours units:    Index 2012=100
    Deflator units: Index 2012=100

Next, compute the quarterly capital stock series for the US using the perpetual inventory method. The discrete-time Solow growth model is given by:

\begin{align}
Y_t & = A_tK_t^{\alpha}L_t^{1-\alpha} \tag{1}\\
C_t & = (1-s)Y_t \tag{2}\\
Y_t & = C_t + I_t \tag{3}\\
K_{t+1} & = I_t + (1-\delta)K_t \tag{4}\\
A_{t+1} & = (1+g)A_t \tag{5}\\
L_{t+1} & = (1+n)L_t \tag{6}.
\end{align}

Here the model is assumed to be quarterly so $n$ is the *quarterly* growth rate of labor hours, $g$ is the *quarterly* growth rate of TFP, and $\delta$ is the *quarterly* rate of depreciation of the capital stock. Given a value of the quarterly depreciation rate $\delta$, an investment series $I_t$, and an initial capital stock $K_0$, the law of motion for the capital stock, Equation (4), can be used to compute an implied capital series. But we don't know $K_0$ or $\delta$ so we'll have to *calibrate* these values using statistics computed from the data that we've already obtained.

Let lowercase letters denote a variable that's been divided by $A_t^{1/(1-\alpha)}L_t$. E.g.,

\begin{align}
y_t = \frac{Y_t}{A_t^{1/(1-\alpha)}L_t}\tag{7}
\end{align}

Then (after substituting consumption from the model), the scaled version of the model can be written as:

\begin{align}
y_t & = k_t^{\alpha} \tag{8}\\
i_t & = sy_t \tag{9}\\
k_{t+1} & = i_t + (1-\delta-n-g')k_t,\tag{10}
\end{align}

where $g' = g/(1-\alpha)$ is the growth rate of $A_t^{1/(1-\alpha)}$. In the steady state:

\begin{align}
k & = \left(\frac{s}{\delta+n+g'}\right)^{\frac{1}{1-\alpha}} \tag{11}
\end{align}

which means that the ratio of capital to output is constant:

\begin{align}
\frac{k}{y} & = \frac{s}{\delta+n+g'} \tag{12}
\end{align}

and therefore the steady state ratio of depreciation to output is:

\begin{align}
\overline{\delta K/ Y} & = \frac{\delta s}{\delta + n + g'} \tag{13}
\end{align}

where $\overline{\delta K/ Y}$ is the long-run average ratio of depreciation to output. We can use Equation (13) to calibrate $\delta$ given $\overline{\delta K/ Y}$, $s$, $n$, and $g'$.

Furthermore, in the steady state, the growth rate of output is constant:

\begin{align}
\frac{\Delta Y}{Y} & = n + g' \tag{14}
\end{align}

1. Assume $\alpha = 0.35$.
2. Calibrate $s$ as the average of ratio of investment to GDP.
3. Calibrate $n$ as the average quarterly growth rate of labor hours.
4. Calibrate $g'$ as the average quarterly growth rate of real GDP minus n.
5. Calculate the average ratio of depreciation to GDP $\overline{\delta K/ Y}$ and use the result to calibrate $\delta$. That is, find the average ratio of Current-Cost Depreciation of Fixed Assets (FRED series ID: M1TTOTL1ES000) to GDP (FRED series ID: GDPA). Then calibrate $\delta$ from the following steady state relationship:
\begin{align}
\delta & = \frac{\left( \overline{\delta K/ Y} \right)\left(n + g' \right)}{s - \left( \overline{\delta K/ Y} \right)} \tag{15}
\end{align}
6. Calibrate $K_0$ by asusming that the capital stock is initially equal to its steady state value:
\begin{align}
K_0 & = \left(\frac{s}{\delta + n + g'}\right) Y_0 \tag{16}
\end{align}

Then, armed with calibrated values for $K_0$ and $\delta$, compute $K_1, K_2, \ldots$ recursively. See Timothy Kehoe's notes for more information on the perpetual inventory method:

http://users.econ.umn.edu/~tkehoe/classes/GrowthAccountingNotes.pdf

```python
# Set the capital share of income
alpha = 0.35

# Average saving rate
s = np.mean(invest.data/gdp.data)

# Average quarterly labor hours growth rate
n = (hours.data[-1]/hours.data[0])**(1/(len(hours.data)-1)) - 1

# Average quarterly real GDP growth rate
g = ((gdp.data[-1]/gdp.data[0])**(1/(len(gdp.data)-1)) - 1) - n

# Compute annual depreciation rate
depA = fp.series('M1TTOTL1ES000')
gdpA = fp.series('gdpa')

gdpA = gdpA.window([gdp.data.index[0],gdp.data.index[-1]])
gdpA,depA = fp.window_equalize([gdpA,depA])

deltaKY = np.mean(depA.data/gdpA.data)
delta = (n+g)*deltaKY/(s-deltaKY)

# print calibrated values:
print('Avg saving rate:        ',round(s,5))
print('Avg annual labor growth:',round(4*n,5))
print('Avg annual gdp growth:  ',round(4*g,5))
print('Avg annual dep rate:    ',round(4*delta,5))

# Construct the capital series. Note that the GPD and investment data are reported on an annualized basis
# so divide by 4 to get quarterly data.
capital = np.zeros(len(gdp.data))
capital[0] = gdp.data[0]/4*s/(n+g+delta)

for t in range(len(gdp.data)-1):
    capital[t+1] = invest.data[t]/4 + (1-delta)*capital[t]

# Save in a fredpy series
capital = fp.to_fred_series(data = capital,dates =gdp.data.index,units = gdp.units,title='Capital stock of the US',frequency='Quarterly')
# plot the computed capital series
plt.plot(capital.data.index,capital.data,'-',lw=3,alpha = 0.7)
plt.ylabel(capital.units)
plt.title(capital.title)
plt.grid()
```

```python
# Compute TFP
tfp = gdp.data/capital.data**alpha/hours.data**(1-alpha)
tfp = fp.to_fred_series(data = tfp,dates =gdp.data.index,units = gdp.units,title='TFP of the US',frequency='Quarterly')

# Plot the computed capital series
plt.plot(tfp.data.index,tfp.data,'-',lw=3,alpha = 0.7)
plt.ylabel(tfp.units)
plt.title(tfp.title)
plt.grid()
```

```python
# Convert each series into per capita using civilian pop 16 and over
gdp = gdp.per_capita(civ_pop=True)
cons = cons.per_capita(civ_pop=True)
invest = invest.per_capita(civ_pop=True)
hours = hours.per_capita(civ_pop=True)
capital = capital.per_capita(civ_pop=True)
m2 = m2.per_capita(civ_pop=True)

# Put GDP, consumption, investment, and M2 in units of thousands of dollars per person
gdp.data = gdp.data*1000
cons.data = cons.data*1000
invest.data = invest.data*1000
capital.data = capital.data*1000
m2.data = m2.data/1000

# Scale hours per person to equal 100 on October (Quarter III) of 2012
hours.data = hours.data/hours.data.loc['2012-10-01']*100

# Compute and plot log real GDP, log consumption, log investment, log hours
gdp_log = gdp.log()
cons_log = cons.log()
invest_log = invest.log()
hours_log = hours.log()
capital_log = capital.log()
tfp_log = tfp.log()
m2_log = m2.log()
m2_log = m2.log()
```

```python
# HP filter to isolate trend and cyclical components
gdp_log_cycle,gdp_log_trend = gdp_log.hp_filter()
cons_log_cycle,cons_log_trend = cons_log.hp_filter()
invest_log_cycle,invest_log_trend = invest_log.hp_filter()
hours_log_cycle,hours_log_trend = hours_log.hp_filter()
capital_log_cycle,capital_log_trend = capital_log.hp_filter()
tfp_log_cycle,tfp_log_trend = tfp_log.hp_filter()
m2_log_cycle,m2_log_trend = m2_log.hp_filter()
tb3mo_cycle,tb3mo_trend = tb3mo.hp_filter()
unemp_cycle,unemp_trend = unemp.hp_filter()
pcec_cycle,pcec_trend = pcec.hp_filter()
```

```python
# Create a DataFrame with actual and trend data
data = pd.DataFrame({
        'gdp':gdp.data,
        'gdp_trend':np.exp(gdp_log_trend.data),
        'gdp_cycle':gdp_log_cycle.data,
        'consumption':cons.data,
        'consumption_trend':np.exp(cons_log_trend.data),
        'consumption_cycle':cons_log_cycle.data,
        'investment':invest.data,
        'investment_trend':np.exp(invest_log_trend.data),
        'investment_cycle':invest_log_cycle.data,
        'hours':hours.data,
        'hours_trend':np.exp(hours_log_trend.data),
        'hours_cycle':hours_log_cycle.data,
        'capital':capital.data,
        'capital_trend':np.exp(capital_log_trend.data),
        'capital_cycle':capital_log_cycle.data,
        'tfp':tfp.data,
        'tfp_trend':np.exp(tfp_log_trend.data),
        'tfp_cycle':tfp_log_cycle.data,
        'real_m2':m2.data,
        'real_m2_trend':np.exp(m2_log_trend.data),
        'real_m2_cycle':m2_log_cycle.data,
        't_bill_3mo':tb3mo.data,
        't_bill_3mo_trend':tb3mo_trend.data,
        't_bill_3mo_cycle':tb3mo_cycle.data,
        'pce_inflation':pcec.data,
        'pce_inflation_trend':pcec_trend.data,
        'pce_inflation_cycle':pcec_cycle.data,
        'unemployment':unemp.data,
        'unemployment_trend':unemp_trend.data,
        'unemployment_cycle':unemp_cycle.data,
    },index = gdp.data.index)

# # RBC Data
# columns_ordered =[]
# names = ['gdp','consumption','investment','hours','capital','tfp']
# for name in names:
#     columns_ordered.append(name)
#     columns_ordered.append(name+'_trend')

# data[columns_ordered].to_csv('../Csv/rbc_data_actual_trend.csv')

# # Create a DataFrame with actual, trend, and cycle data
# columns_ordered =[]
# names = ['gdp','consumption','investment','hours','capital','tfp']
# for name in names:
#     columns_ordered.append(name)
#     columns_ordered.append(name+'_trend')
#     columns_ordered.append(name+'_cycle')

# data[columns_ordered].to_csv('../Csv/rbc_data_actual_trend_cycle.csv')

# Business Cycle Data
columns_ordered =[]
names = ['gdp','consumption','investment','hours','capital','tfp','real_m2','t_bill_3mo','pce_inflation','unemployment']
for name in names:
    columns_ordered.append(name)
    columns_ordered.append(name+'_trend')

data[columns_ordered].to_csv('../Csv/business_cycle_data_actual_trend.csv')

# Create a DataFrame with actual, trend, and cycle data
columns_ordered =[]
names = ['gdp','consumption','investment','hours','capital','tfp','real_m2','t_bill_3mo','pce_inflation','unemployment']
for name in names:
    columns_ordered.append(name)
    columns_ordered.append(name+'_trend')
    columns_ordered.append(name+'_cycle')

data[columns_ordered].to_csv('../Csv/business_cycle_data_actual_trend_cycle.csv')
```


# Linear Regression

```
# Linear Regression 1 Variable
# Independent: Number of A grades in 1st year (freshmen year)
x = [3,1,0,4]
# Dependent: Number of A grades in 2nd year (sophomore year)
y = [2,2,1,3]
# Hypothesis function
def hypothesis(theta0,theta1,x):
  h = theta0 + theta1 * x
  return h

# Cost function
def cost(theta0,theta1,x,y):
  # Number of training examples
  m = len(x)
  J = 0
  for i in range(m):
    J = J + ( hypothesis(theta0,theta1,x[i]) - y[i] )**2
    # print(i, " : ", J)
  J = J / (2*m)
  return J

theta0 = 0
theta1 = 1
J = cost(theta0,theta1,x,y)
print(J)

theta0 = -1
theta1 = 2
theta1 = 0.5
x = 6
x = 4
h = hypothesis(theta0,theta1,x)
print(h)

x = [1,2,4,0]
y = [0.5,1,2,0]
v_theta0 = [1,1,0.5,0,0.5]
v_theta1 = [0.5,1,0,0.5,0.5]
for i in range(len(v_theta0)):
  J = cost(v_theta0[i],v_theta1[i],x,y)
  print(i, " : ", J)
```

    0.5
    1.0
    0  :  0.5
    1  :  2.03125
    2  :  0.34375
    3  :  0.0
    4  :  0.125

# Linear Algebra

```
import numpy as np
import scipy.linalg as la
from numpy.linalg import matrix_power as mpow

M = np.array([[3,4],[-1,5]])
print('Matrix:\n', M)
print('Matrix Transposition:\n', M.T)
print('Elementwise Multiplication:\n', M * M)
print('Matrix Multiplication:\n', M @ M)
print('Matrix Power:\n', mpow(M,2))
I = np.eye(2)
print('Identity:\n', I)
A = np.array([[1,2],[3,4]])
print('Matrix:\n', A)
print('Matrix Invertion:\n', la.inv(A))
trace_A = np.trace(A)
print('Trace:\n', trace_A)
det_A = la.det(A)
print('Determinant:\n', det_A)
print('Hamilton:\n +A**2 = \n', A @ A)
print('trace_A * A = \n', trace_A * A)
print('det_A * I = \n', det_A * I)
print('A**2 - trace_A * A + det_A * I = \n', A @ A - trace_A * A + det_A * I)

A = np.array([[4,3],[6,9]])
B = np.array([[-2,9],[-5,2]])
print('A = \n', A)
print('B = \n', B)
print('A-B = \n', A-B)

x = np.array([2,7,4,1])
print('x = \n', x)
print('3*x = \n', 3*x)

u = np.array([5,1,9])
print('u = \n', u)
print('uT = \n', u.reshape(3,1))

u = np.array([1,3,-1])
v = np.array([2,2,4])
print('u = \n', u)
print('v = \n', v)
print('uT * v = \n', u @ v)
```

    Matrix:
     [[ 3  4]
     [-1  5]]
    Matrix Transposition:
     [[ 3 -1]
     [ 4  5]]
    Elementwise Multiplication:
     [[ 9 16]
     [ 1 25]]
    Matrix Multiplication:
     [[ 5 32]
     [-8 21]]
    Matrix Power:
     [[ 5 32]
     [-8 21]]
    Identity:
     [[1. 0.]
     [0. 1.]]
    Matrix:
     [[1 2]
     [3 4]]
    Matrix Invertion:
     [[-2.   1. ]
     [ 1.5 -0.5]]
    Trace:
     5
    Determinant:
     -2.0
    Hamilton:
     +A**2 =
     [[ 7 10]
     [15 22]]
    trace_A * A =
     [[ 5 10]
     [15 20]]
    det_A * I =
     [[-2. -0.]
     [-0. -2.]]
    A**2 - trace_A * A + det_A * I =
     [[0. 0.]
     [0. 0.]]
    A =
     [[4 3]
     [6 9]]
    B =
     [[-2  9]
     [-5  2]]
    A-B =
     [[ 6 -6]
     [11  7]]
    x =
     [2 7 4 1]
    3*x =
     [ 6 21 12  3]
    u =
     [5 1 9]
    uT =
     [[5]
     [1]
     [9]]
    u =
     [ 1  3 -1]
    v =
     [2 2 4]
    uT * v =
     4

$$ h_\theta(x) = \theta_0 + \theta_1 x_1 + ... + \theta_n x_n = \theta^Tx $$

\begin{align*}h_\theta(x) =\begin{bmatrix}\theta_0 \hspace{2em} \theta_1 \hspace{2em} ... \hspace{2em} \theta_n\end{bmatrix}\begin{bmatrix}x_0 \newline x_1 \newline \vdots \newline x_n\end{bmatrix}= \theta^T x\end{align*}

\begin{align*}
x_j^{(i)} &= \text{value of feature } j \text{ in the }i^{th}\text{ training example}
\newline x^{(i)}& = \text{the input (features) of the }i^{th}\text{ training example}
\newline m &= \text{the number of training examples}
\newline n &= \text{the number of features} \end{align*}

\begin{align*}& \text{repeat until convergence:} \; \lbrace \newline \; & \theta_j := \theta_j - \alpha \frac{1}{m} \sum\limits_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) \cdot x_j^{(i)} \; & \text{for j := 0...n}\newline \rbrace\end{align*}

feature scaling and mean normalization

$$ x_i := \frac{x_i-\mu_i}{s_i} $$

If $\alpha$ is too small: slow convergence.

If $\alpha$ is too large: may not decrease on every iteration and thus may not converge.

```
x0 = [1,1,1,1]
x1 = [2104,1416,15314,852]
x2 = [5,3,3,2]
x3 = [1,2,2,1]
x4 = [45,40,30,36]
y = [46,232,315,178]
m = len(y)
x0 = [1,1,1,1]

X = [x0,x1,x2,x3,x4]
```

# Normal Equation
$$ \theta = (X^TX)^{-1}X^Ty $$

$ (X^TX)$ non invertible, singular degenerate

1.   Redundant features (LD)
Delete one of such futures

2.   Too many features (e.g. $ m \leq n$ )

Delete some futures or use regularization

# Gradient Descent vs Normal Equation

| Criteria | Gradient Descent | Normal Equation   |
|---|------|------|
| Need to choose alpha |   Yes  | No |
| Iterations |   Many  | Single |
| Time |   $O(kn^2)$  | $O(n^3)$ |
| Size |   Large  | Not very large <10.000 |

# Classification
Binary: 0 Negative class, 1 Positive class
Classification: $ y=0 $
 or $1$

Multiclasse: >2 classes

## Linear regression

$$ h_\theta(x) = \theta^Tx $$

if $ h_\theta(x) \geq 0.5 $, predict $ y = 1$

if $ h_\theta(x) \lt 0.5 $, predict $ y = 0$

$ h_\theta(x) < 0 $ or $  > 1 $

Conclusion:

Linear regression doesn't suit classification problems very well

## Logistic Regression
One desires that
$$ 0 \leq h_\theta(x) \leq 1 $$

We have the sigmoid or logistic function
$$ g(z) = \frac{1}{1+e^{-z}} $$

With the following properties
\begin{align*}
g(z) \geq 0.5, \qquad  \text{when} \; z \geq 0
\end{align*}
\begin{align*}
&z=0, \qquad\quad e^{0}=1 &\Rightarrow g(z)=\frac{1}{2} \newline
&z \to \infty,\qquad\; e^{-\infty} \to 0 &\Rightarrow g(z)=1 \newline
&z \to -\infty,\quad\;\; e^{\infty}\to \infty &\Rightarrow g(z)=0
\end{align*}

```
import numpy as np
import matplotlib.pyplot as plt
# evenly sampled z at 0.1 intervals
z = np.arange(-5, 5., 0.1)
exp_z = np.exp(z)
exp_minus_z = np.exp(-z)
one_plus_exp_minus_z = 1 + np.exp(-z)
inv_one_plus_exp_minus_z = 1./(1 + np.exp(-z))

plt.style.use(['dark_background'])
plt.rcParams.update({'font.size': 15})

f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
ax1.plot(z, exp_z)
ax1.set_title('$e^{z}$')
ax2.scatter(z, exp_minus_z)
ax2.set_title('$e^{-z}$')
#ax2.set_title('$\frac{1}{1+e^{-z}}$')
f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
ax1.plot(z, one_plus_exp_minus_z)
ax1.set_title('$1 + e^{-z}$')
ax2.scatter(z, inv_one_plus_exp_minus_z)
ax2.set_title('$\\frac{1}{1+e^{-z}}$')
```

Hypothesis representation
$$ h_\theta(x) = g(\theta^Tx) = \frac{1}{1+e^{-\theta^Tx}} $$

Thus
\begin{align*}
h_\theta(x) = g(\theta^T x) \geq 0.5, \qquad \text{when} \; \theta^T x \geq 0
\end{align*}
In a nutshell
\begin{align*}& \theta^T x \geq 0 \Rightarrow y = 1 \newline& \theta^T x < 0 \Rightarrow y = 0 \newline\end{align*}

Probability that $y=1$, given $x$ parametrized by $\theta$

$$h_\theta(x) = P(y=1|x;\theta) $$

Regarding that
$$  P(y=0) + P(y=1) = 1 $$

### Decision Boundary

if $ h_\theta(x) \geq 0.5 $,
predict $ y = 1$;
$ g(z) \geq 0.5 $,
when $z = \theta^Tx \geq 0$

if $ h_\theta(x) \lt 0.5 $,
predict $ y = 0$
$ g(z) \lt 0.5 $,
when $z = \theta^Tx \lt 0$

Decision boundary is a property of the hypothesis and not of the dataset

### Cost function

\begin{align*}
& J(\theta) = \dfrac{1}{m} \sum_{i=1}^m \mathrm{Cost}(h_\theta(x^{(i)}),y^{(i)}) \newline
& \mathrm{Cost}(h_\theta(x),y) = -\log(h_\theta(x)) \; & \text{if y = 1} \newline
& \mathrm{Cost}(h_\theta(x),y) = -\log(1-h_\theta(x)) \; & \text{if y = 0}
\end{align*}

\begin{align*}
& \mathrm{Cost}(h_\theta(x),y) = 0 \text{ if } h_\theta(x) = y \newline
& \mathrm{Cost}(h_\theta(x),y) \rightarrow \infty \text{ if } y = 0 \; \mathrm{and} \; h_\theta(x) \rightarrow 1 \newline
& \mathrm{Cost}(h_\theta(x),y) \rightarrow \infty \text{ if } y = 1 \; \mathrm{and} \; h_\theta(x) \rightarrow 0
\end{align*}

Penalizes absence of extremely high and low probability cases (as 0 and 1)

```
import numpy as np
import matplotlib.pyplot as plt
# evenly sampled z at 0.1 intervals
z = np.arange(0.01, 1., 0.01)

log_z = np.log(z)
negative_log_1minusz = -np.log(1-z)
#fig, ax = plt.subplots()
# red dashes, blue squares and green triangles
plt.style.use(['dark_background'])
plt.rcParams.update({'font.size': 15})
plt.plot(z, -log_z, 'bs', z, negative_log_1minusz, 'g^')
plt.legend(['$-\log(z)$, if $y=1$','$-\log(1-z)$, if $y=0$'])
plt.xlabel('Hypothesis: $z = h_\\theta(x) $')
plt.ylabel('Cost Function: $J(h_\\theta(x),y) $')
#plt.axis([0, 1, 0, 5])
plt.show()

f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
ax1.plot(z, log_z)
ax1.set_title('$\log(z)$')
ax2.scatter(z, -log_z)
ax2.set_title('$-\log(z)$');

```

As always $y=0$ or $y=1$
$$ \mathrm{Cost}(h_\theta(x),y) = -y \log(h_\theta(x)) - (1-y)\log(1-h_\theta(x)) $$

Principle of maximum likelihood estimation: statistics idea to find parameters

\begin{align*}
& h = g(X\theta)\newline
& J(\theta) = \frac{1}{m} \cdot \left(-y^{T}\log(h)-(1-y)^{T}\log(1-h)\right) \end{align*}

\begin{align*}
& \text{Repeat} \; \lbrace \newline
& \; \theta_j := \theta_j - \frac{\alpha}{m} \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)}) x_j^{(i)} \newline
& \rbrace
\end{align*}

$$ \theta := \theta - \frac{\alpha}{m}X^T(g(X\theta)-y)$$

```

```

Optimization algorithms (Line search)
*   Gradient descent
*   Conjugate gradient
*   BFGS
*   L-BFGS

Advantages
*   No need to manually pick $\alpha$
*   Often faster than gradient descent

Disadvantages
*   More complex

Compute cost function and its derivative

```
def cost_function(theta):
  J = (theta[0] - 5) ** 2 + (theta[1] - 5) ** 2
  return J
def derivative_cost_function(theta):
  dJdtheta = [2*(theta[0] - 5),2*(theta[1] - 5)]
  return dJdtheta
theta = [5, 5]
J = cost_function(theta)
dJdtheta = derivative_cost_function(theta)
print(J)
print(dJdtheta)
```

    34
    [10, 6]

# Optimization Algorithms

```
>>> from scipy.optimize import minimize, rosen, rosen_der
>>> x0 = [1.3, 0.7, 0.8, 1.9, 1.2]
```

## Nelder Mead

```
res = minimize(rosen, x0, method='Nelder-Mead',
               options={'gtol': 1e-6, 'disp': True})
print(res.x)
print(res.message)
```

    Optimization terminated successfully.
             Current function value: 0.000066
             Iterations: 141
             Function evaluations: 243
    [0.99910115 0.99820923 0.99646346 0.99297555 0.98600385]
    Optimization terminated successfully.

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: OptimizeWarning: Unknown solver options: gtol

## BFGS

```
res = minimize(rosen, x0, method='BFGS', jac=rosen_der,
...                options={'gtol': 1e-6, 'disp': True})
print(res.x)
print(res.hess_inv)
```

    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 26
             Function evaluations: 31
             Gradient evaluations: 31
    [1. 1. 1. 1. 1.]
    [[0.00763624 0.0125159  0.02359808 0.0465684  0.09317974]
     [0.0125159  0.02489687 0.04727754 0.09353774 0.18708048]
     [0.02359808 0.04727754 0.09483095 0.18783847 0.37561493]
     [0.0465684  0.09353774 0.18783847 0.37715312 0.75414172]
     [0.09317974 0.18708048 0.37561493 0.75414172 1.51296479]]

```
fun = lambda x: (x[0] - 1)**2 + (x[1] - 2.5)**2
>>> cons = ({'type': 'ineq', 'fun': lambda x:  x[0] - 2 * x[1] + 2},
...         {'type': 'ineq', 'fun': lambda x: -x[0] - 2 * x[1] + 6},
...         {'type': 'ineq', 'fun': lambda x: -x[0] + 2 * x[1] + 2})
>>> bnds = ((0, None), (0, None))
```

SLSQP

```
>>> res = minimize(fun, (2, 0), method='SLSQP', bounds=bnds,
...                constraints=cons, options={'gtol': 1e-6, 'disp': True})
print(res.x)
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.8000000011920985
                Iterations: 3
                Function evaluations: 13
                Gradient evaluations: 3
    [1.4 1.7]

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: OptimizeWarning: Unknown solver options: gtol

# Multiclass
One vs all/rest

Train a logistic regression classifier $h^{(i)}_\theta(x)$ for each class $i$ to predict the probability that $y = i$.

To make a prediction on a new $x$,
pick the class $i$
that maximizes $h_\theta(x)$

\begin{align*}
& y \in \lbrace0, 1 ... n\rbrace \newline
& h_\theta^{(0)}(x) = P(y = 0 | x ; \theta) \newline
& h_\theta^{(1)}(x) = P(y = 1 | x ; \theta) \newline
& \cdots \newline
& h_\theta^{(n)}(x) = P(y = n | x ; \theta) \newline
& \mathrm{prediction} = \max_i( h_\theta ^{(i)}(x) )
\end{align*}

2.Question 2
Suppose you have the following training set, and
fit a logistic regression classifier
$$ h_\theta(x) = g(\theta_0 + \theta_1x_1 + \theta_2 x_2)$$

TRUE
*   $J(\theta)$ will be a convex function, so gradient descent should converge to the global minimum.
*   At the optimal value of \thetaθ (e.g., found by fminunc), we will have $J(\theta) \geq 0$.
*   Adding polynomial features (e.g., instead using
$$h_\theta(x) = g(\theta_0 + \theta_1x_1 + \theta_2 x_2 + \theta_3 x_1^2 + \theta_4 x_1 x_2 + \theta_5 x_2^2) $$
could increase how well we can fit the training data,
but wouldn't increase $J(\theta)$ because we are now summing over more terms.

FALSE

The positive and negative examples cannot be separated using a straight line.
So, gradient descent will fail to converge.

FALSE

Because the positive and negative examples cannot be separated using a straight line,
linear regression will perform as well as logistic regression on this data.

FALSE

If we train gradient descent for enough iterations, for some examples $x^{(i)}$
in the training set it is possible to obtain $h_\theta(x^{(i)}) > 1$.

3.Question 3
For logistic regression, the gradient is given by
$$\frac{\partial}{\partial \theta_j} J(\theta)
= \frac{1}{m}\sum_{i=1}^m{ (h_\theta(x^{(i)}) - y^{(i)}) x_j^{(i)}}$$
Which of these is a correct gradient descent update for logistic regression
with a learning rate of $\alpha$?

FALSE
$$\theta := \theta - \alpha \frac{1}{m} \sum_{i=1}^m{
 \left(\frac{1}{1 + e^{-\theta^T x^{(i)}}} - y^{(i)}\right) x^{(i)}}$$

FALSE (the hypothesis function is not like that)
$$\theta := \theta - \alpha \frac{1}{m} \sum_{i=1}^m{
 \left(\theta^T x - y^{(i)}\right) x^{(i)}}, \quad
 \text{(simultaneously update for all j)}$$

TRUE (vectorized)
$$\theta := \theta - \alpha \frac{1}{m} \sum_{i=1}^m{
 (h_\theta(x^{(i)}) - y^{(i)}) x^{(i)}}, \quad
 \text{(simultaneously update for all j)}$$

TRUE (discrete)
$$\theta_j := \theta_j - \alpha \frac{1}{m} \sum_{i=1}^m{
 \left(\theta^T x - y^{(i)}\right) x_j^{(i)}}, \quad
 \text{(simultaneously update for all j)}$$

4.Question 4

TRUE
*   The cost function $J(\theta)$ for logistic regression trained with $m \geq 1$
examples is always greater than or equal to zero.
*   The one-vs-all technique allows you to use logistic regression
for problems in which each $y^{(i)}$ comes from a fixed, discrete set of values.

FALSE (train k classifiers for k classes)

Since we train one classifier when there are two classes,
we train two classifiers when there are three classes (and we do one-vs-all classification).

FALSE (that's not the reason)

For logistic regression, sometimes gradient descent will converge to a local minimum
(and fail to find the global minimum).
This is the reason we prefer more advanced optimization algorithms
such as fminunc (conjugate gradient/BFGS/L-BFGS/etc).

5.Question 5
Suppose you train a logistic classifier
$$h_\theta(x) = g(\theta_0 + \theta_1x_1 + \theta_2 x_2)$$
Suppose
$$\theta_0 = 6, \theta_1 = -1, \theta_2 = 0 $$
Decision boundary is

|		 |     |        |
|---|-----------|------|
|$y = 1$| 	|	$y = 0$|
|		    | $x1 = 6$  |        |

# Overfitting

Underfit: high bias

Overfit:
*   high variance
*   too many features
*   fit the training set very well
*   fail to new generalized cases

Prevent overfitting
1.   Reduce number of features
*   Manual selection
*   Algorithm selection

2.   Regularization
*   Keep all features, but reduce magnitude of parameters $\theta_j$
*   Work well, when a lot of slightly useful features
* Too large $\lambda$ can cause underfit
* $J(\theta)$ is still convex
* can impair the performance on training examples
* $\geq$ performance on predictions not in the training set

## Regularization

Cost function

Penalization

$$ \min_\theta =
\dfrac{1}{2m} \sum_{i=1}^m (h_\theta(x^{(i)})-y^{(i)})^2
+ \lambda \sum_{j=1}^n \theta_j^2 $$

### Gradient descent

\begin{align*} & \text{Repeat}\ \lbrace \newline & \ \ \ \ \theta_0 := \theta_0 - \alpha\ \frac{1}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_0^{(i)} \newline & \ \ \ \ \theta_j := \theta_j - \alpha\ \left[ \left( \frac{1}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} \right) + \frac{\lambda}{m}\theta_j \right] &\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline & \rbrace \end{align*}

$$ \theta_j := \theta_j \left(1- \alpha \frac{\lambda}{n}\right)
- \frac{\alpha}{m} \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)}) x_j^{(i)} $$

$\lambda, \alpha, m > 0 \implies 1- \alpha \frac{\lambda}{m} < 1 $

### Normal equation

\begin{align*}& \theta = \left( X^TX + \lambda \cdot L \right)^{-1} X^Ty \newline& \text{where}\ \ L = \begin{bmatrix} 0 & & & & \newline & 1 & & & \newline & & 1 & & \newline & & & \ddots & \newline & & & & 1 \newline\end{bmatrix}\end{align*}

$X$ is non-invertible if $m < n$, and may be non-invertible if $m = n$.

However $X^TX + \lambda \cdot L$ becomes invertible.

Logistic Regression
$$ J(\theta) = -\frac{1}{m}\sum_{i=1}^m \left[
  y^{(i)}\log(h_\theta(x^{(i)}))
  +(1-y^{(i)})\log(1-h_\theta(x^{(i)}))
  \right] $$

Regularized Logistic Regression
$$ J(\theta) = -\frac{1}{m}\sum_{i=1}^m \left[
  y^{(i)}\log(h_\theta(x^{(i)}))
  +(1-y^{(i)})\log(1-h_\theta(x^{(i)}))
  \right]
  + \frac{\lambda}{2m} \sum_{j=1}^n \theta_j^2 $$

Hypothesis function
$$ h_\theta(x) = \frac{1}{1+e^{-\theta^Tx}} $$

# Linear vs Logistic Regression

## Differences
\begin{align*}
&\text{Regression} & \text{Linear} & & \text{Logistic}\newline
&h = & \theta^T X && \dfrac{1}{1+e^{-\theta^T X}} \newline
&J = & \frac{(h-y)^T(h-y)}{2m} &&\frac{\left(-y^{T}\log(h)-(1-y)^{T}\log(1-h)\right)}{m}
\end{align*}

## Similarities
Gradient
$$\frac{\partial J}{\partial \theta}=  \frac{X^{T}(h-y)}{m} $$
Cost Function Penalty
$$pJ = \frac{\lambda}{2m} \sum_{j=1}^n \theta_j^2$$
Gradient Function Penalty
$$p\frac{\partial J}{\partial \theta} = \frac{\lambda}{m} \sum_{j=1}^n \theta_j$$

Remark
$\theta_0$ is not penalized

# Parameter estimation

Gradient Descent
$$ \theta := \theta - \alpha \frac{\partial J}{\partial \theta} $$

Normal Equation
$$ \theta = (X^TX)^{-1}X^Ty $$

# Neural Networks

* gray scale: 0-255

* RGB: 3x 0-255

1980 popular, ressuregence today

Dendrite
* input wires
Axon
* output wires
Synapse

Sigmoid Logistic activation function
$$ h_\Theta(x) = \frac{1}{1+e^{-\Theta^Tx}} $$

Bias
* $x_0 = 1$

Parameters
* weights

Layer
1. Input
2. Hidden
3. Output

$a_i^{(j)}$: Activation of unit $i$ in layer $j$

$\Theta^{(j)}$: matrix of weights controlling function mapping from layer $j$ to layer $j+1$

$s_j$ units in layer $j$, $s_{j+1}$ units in layer ${j+1}$
$\implies \Theta^{(j)} \in \mathbb{R}^{ s_{j+1} \times (s_{j}+1)}$

where $1$ refers to the bias nodes $x_0$ and $\Theta_0^{(j)}$

## Forward propagation

Ex: $1$ hidden layer
$$[x_0 x_1 x_2 x_3] \implies [a_1^{(2)} a_2^{(2)} a_3^{(2)}] \implies  h_\Theta(x) $$

Activation functions
\begin{align*}
a_1^{(2)} = g(\Theta_{10}^{(1)}x_0 + \Theta_{11}^{(1)}x_1 + \Theta_{12}^{(1)}x_2 + \Theta_{13}^{(1)}x_3) = g(z_1^{(2)}) \newline
a_2^{(2)} = g(\Theta_{20}^{(1)}x_0 + \Theta_{21}^{(1)}x_1 + \Theta_{22}^{(1)}x_2 + \Theta_{23}^{(1)}x_3) = g(z_2^{(2)}) \newline
a_3^{(2)} = g(\Theta_{30}^{(1)}x_0 + \Theta_{31}^{(1)}x_1 + \Theta_{32}^{(1)}x_2 + \Theta_{33}^{(1)}x_3) = g(z_3^{(2)}) \newline
h_\Theta(x) = a_1^{(3)} = g(\Theta_{10}^{(2)}a_0^{(2)} + \Theta_{11}^{(2)}a_1^{(2)} + \Theta_{12}^{(2)}a_2^{(2)} + \Theta_{13}^{(2)}a_3^{(2)})
\end{align*}

Vectorize
\begin{align*}
z_1^{(2)} = \Theta_{10}^{(1)}x_0 + \Theta_{11}^{(1)}x_1 + \Theta_{12}^{(1)}x_2 + \Theta_{13}^{(1)}x_3 \newline
z_2^{(2)} = \Theta_{20}^{(1)}x_0 + \Theta_{21}^{(1)}x_1 + \Theta_{22}^{(1)}x_2 + \Theta_{23}^{(1)}x_3 \newline
z_3^{(2)} = \Theta_{30}^{(1)}x_0 + \Theta_{31}^{(1)}x_1 + \Theta_{32}^{(1)}x_2 + \Theta_{33}^{(1)}x_3
\end{align*}

\begin{align*}
z_k^{(2)} = \Theta_{k0}^{(1)}x_0 + \Theta_{k1}^{(1)}x_1 + \cdot\cdot\cdot + \Theta_{kn}^{(1)}x_n
\end{align*}

\begin{align*}
a_1^{(2)} = g(z_1^{(2)}) \newline
a_2^{(2)} = g(z_2^{(2)}) \newline
a_3^{(2)} = g(z_3^{(2)})
\end{align*}

\begin{align*}
a^{(1)} =
x =
\begin{bmatrix}
x_0 \newline
x_1 \newline
\cdots \newline
x_n
\end{bmatrix}
,\qquad
a^{(j)} =
g(z^{(j)}) =
g\left(
\begin{bmatrix}
z_1^{(j)} \newline
z_2^{(j)} \newline
\cdots \newline
z_n^{(j)}
\end{bmatrix}
\right)
\end{align*}

$$ z^{(j)} = \Theta^{(j-1)}a^{(j-1)} $$

$$ z^{(j+1)} = \Theta^{(j)}a^{(j)} $$

$$ h_\Theta(x) = a^{(j+1)} = g(z^{(j+1)}) $$

Forward propagation algorithm
\begin{align}
a^{(1)} &= x \quad (\text{add } a_0^{(1)}) \\
z^{(2)} &= \Theta^{(1)}a^{(1)}\\
a^{(2)} &= g(z^{(2)}) \quad (\text{add } a_0^{(2)}) \\
z^{(3)} &= \Theta^{(2)}a^{(2)}\\
a^{(3)} &= g(z^{(3)}) \quad (\text{add } a_0^{(3)}) \\
z^{(4)} &= \Theta^{(3)}a^{(3)}\\
a^{(4)} &= g(z^{(4)})
\end{align}

```

x = [1;2;3];
Theta1 = ones(3,3);
a2 = zeros(3,1);

function sig = sigmoid(z)
  sig = 1/(1 + exp(-z))
end

for i =1:3
  for j = 1:3
    a2(i) = a2(i) + x(j) * Theta1(i,j);
  end
  a2(i) = sin(a2(i));
end

a2v = sin(Theta1*x)

a2v = sin(x * Theta1)

t1 = [1 2.1 1.3; 1 0.6 -1.2]
t1_ = [1 0.6 -1.2; 1 2.1 1.3]
t2 = [1 3.1 4.5]
t2_ = [1 4.5 3.1]

a1 = x
a2 = sigmoid(t1 * a1)
a2_ = sigmoid(t1_ * a1)

a3 = sigmoid(t2 * a2)
a3_ = sigmoid(t2_ * a2_)

h = a3
h = a3_

```

## Non-linear classification: XOR/XNOR

\begin{align*}
y = \;& x_1 \text{XOR}\; x_2 \newline
& x_1 \text{XNOR}\; x_2\newline
& \text{NOT} (x_1 \text{XOR}\; x_2) \newline
\end{align*}
$$ x_1,x_2 \in \{0,1\} $$

\begin{align*}
y = \;& x_1 \text{AND}\; x_2 \newline
\end{align*}

Truth tables

| $x_1$ | $x_2$ |  OR  | AND | XOR | XNOR | NAND | NOR |
|-------|-------|------|-----|-----|------|------|-----|
|1      |1      |1     |1    |0    |1     |0     |0    |
|1      |0      |1     |0    |1    |0     |1     |0    |
|0      |1      |1     |0    |1    |0     |1     |0    |
|0      |0      |0     |0    |0    |1     |1     |1    |

|~$x_1$ |~$x_2$ |  OR  | AND | XOR | XNOR | NAND | NOR |
|-------|-------|------|-----|-----|------|------|-----|
|0      |0      |0     |0    |0    |1     |1     |1    |
|0      |1      |1     |0    |1    |0     |1     |0    |
|1      |0      |1     |0    |1    |0     |1     |0    |
|1      |1      |1     |1    |0    |1     |0     |0    |

* X: exclusive
* N: not
* XNOR: not XOR
* $x_1$ NOR $x_2$  = (NOT $x_1$) AND (NOT $x_2$)

$$\Theta_0^{(1)} = 10$$
Not
$$\Theta_0^{(1)} = -20$$

|                | $x_1$ OR $x_2$ | $x_1$ AND $x_2$ | $x_1$ NOR $x_2$ |
|----------------|:-:|:-:|:-:|
|$\Theta_0^{(1)}$|10 |-30|10 |
|$\Theta_1^{(1)}$|20 |20 |-20|
|$\Theta_2^{(1)}$|20 |20 |-20|

```
import numpy as np
import matplotlib.pyplot as plt
# evenly sampled z at 0.1 intervals
z = np.arange(-5, 5., 0.1)

def sigmoid_function(z):
  sigmoid = 1./(1 + np.exp(-z))
  return sigmoid

inv_one_plus_exp_minus_z = 1./(1 + np.exp(-z))

plt.style.use(['dark_background'])
plt.rcParams.update({'font.size': 15})

plt.scatter(z, sigmoid_function(z))
plt.title('$g(z) = \\dfrac{1}{1+e^{-z}}$')
plt.xlabel('z')

print('g(-4) = ', sigmoid_function(-4))
print('g(4) = ', sigmoid_function(4))

theta = np.array([-10,20,20])
x0 = np.array([1,1,1,1])
x1 = np.array([1,1,0,0])
x2 = np.array([1,0,1,0])
z = theta[0] * x0 + theta[1] * x1 + theta[2] * x2
sigmoid_function(z)
```

## Multiclass Classification

* 10 classes
* 3 layers
* hidden layer has 5 units

one vs all
* $\Theta^{(2)}$ has 60 elements
 * must add 1 element from bias unit

## Quiz

* without hidden layer is not possible to represent XOR function
* any logical binary function can be approximated by some neural network
* sigmoid activation function $\in [0,1]$
* $a_1^{(3)} + a_2^{(3)} + a_3^{(3)} = 1 $

## Backpropagation
* $L$ = total number of layers in the network
* $s_l$ = number of units (not counting bias unit) in layer $l$
* $K$ = number of output units/classes

\begin{gather*}
J(\Theta) =
 - \frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K
 \left[y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k)
 + (1 - y^{(i)}_k)\log (1 - (h_\Theta(x^{(i)}))_k)\right]
 + \frac{\lambda}{2m}\sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}}
 ( \Theta_{j,i}^{(l)})^2
 \end{gather*}

Backpropagation

Given training set
$\{(x^{(1)},y^{(1)}),\cdot\cdot\cdot,(x^{(m)},y^{(m)})\}$
* Set
$$\Delta_{ij}^{(l)}:=0 \quad \forall (l,i,j)$$

For training example $t=1:m$:

1.   Set
$$a^{(1)}:= x^{(t)} \quad (\text{add } a_0^{(1)})$$
2.   Perform forward propagation to compute $a^{(l)}$ for $l=2,\cdot\cdot\cdot,L$
\begin{align}
z^{(2)} &= \Theta^{(1)}a^{(1)}\\
a^{(2)} &= g(z^{(2)}) \quad (\text{add } a_0^{(2)}) \\
z^{(3)} &= \Theta^{(2)}a^{(2)}\\
a^{(3)} &= g(z^{(3)}) \quad (\text{add } a_0^{(3)}) \\
z^{(4)} &= \Theta^{(3)}a^{(3)}\\
a^{(4)} &= g(z^{(4)})
\end{align}
3. Compute  the error
$$\delta^{(L)} = a^{(L)} - y^{(t)}$$
4. Compute $\delta^{(L-1)},\delta^{(L-2)},\cdot\cdot\cdot,\delta^{(2)}$
$$ \delta^{(l)} = ((\Theta^{(l)})^T\delta^{(l-1)}) \;.*\;
a^{(l)} \;.*\; (1-a^{(l)}) $$
where
$$ g'(z^{(l)}) = a^{(l)} \;.*\; (1-a^{(l)}) $$
5. Update $$ \Delta^{(l)}:= \Delta^{(l)} + \delta^{(l+1)}(a^{(l)})^T $$
6. Compute the derivatives
\begin{align}
D_{ij}^{(l)} &:= \dfrac{1}{m}\left(\Delta_{ij}^{(l)} + \lambda\Theta_{ij}^{(l)}\right) &\text{if } j \neq 0 \\
D_{ij}^{(l)} &:= \dfrac{1}{m}\left(\Delta_{ij}^{(l)}\right)
&\text{if } j = 0
\end{align}
where
$$ \dfrac{\partial J(\Theta)}{\partial \Theta_{ij}^{(l)}} = D_{ij}^{(l)}
$$

Unroll

Matrix to vector
``` octave
thetaVector = [ Theta1(:); Theta2(:); Theta3(:); ]
deltaVector = [ D1(:); D2(:); D3(:) ]
```

Vector to matrix
``` octave
Theta1 = reshape(thetaVector(1:110),10,11)
Theta2 = reshape(thetaVector(111:220),10,11)
Theta3 = reshape(thetaVector(221:231),1,11)
```

Gradient Checking

To ckeck if backprop is correctly implemented

$$ \dfrac{\partial J(\Theta)}{\partial \Theta} \approx
\dfrac{J(\Theta+\epsilon)-J(\Theta-\epsilon)}{2\epsilon} $$

$$ \dfrac{\partial J(\Theta)}{\partial \Theta_{j}} \approx
\dfrac{J(\Theta_1,...,\Theta_j+\epsilon,...,\Theta_n)-J(\Theta_1,...,\Theta_j-\epsilon,...,\Theta_n)}{2\epsilon} $$

$ \epsilon = 10^{-4} $

```
x = 1
eps = 0.01

def cost(x):
  J = x ** 3
  return J

def cds(x,eps):
  dJdx = ( cost(x+eps) - cost(x-eps) ) / (2*eps)
  return dJdx

grad_approx = cds(x,eps)
grad_approx
```

    3.0001000000000055

```
x = 1
eps = 0.01

def cost(x):
  J = 2 * x ** 4 + 2
  return J

def cds(x,eps):
  dJdx = ( cost(x+eps) - cost(x-eps) ) / (2*eps)
  return dJdx

grad_approx = cds(x,eps)
grad_approx
```

    8.000800000000007

``` octave
epsilon = 1e-4;
for i = 1:n,
  thetaPlus = theta;
  thetaPlus(i) += epsilon;
  thetaMinus = theta;
  thetaMinus(i) -= epsilon;
  gradApprox(i) = (J(thetaPlus) - J(thetaMinus))/(2*epsilon)
end;
```

Random initialization

Symmetry breaking

``` octave
Theta1 = rand(10,11) * (2 * INIT_EPSILON) - INIT_EPSILON;
Theta2 = rand(10,11) * (2 * INIT_EPSILON) - INIT_EPSILON;
Theta3 = rand(1,11) * (2 * INIT_EPSILON) - INIT_EPSILON;
```

$$ \epsilon_{init} = \dfrac{\sqrt{6}}{\sqrt{L_{in} + L_{out}}} $$
where $L_{in}=s_l$ and $L_{out}=s_{l+1}$ are the # units in the layers adjacent to $\Theta^{(l)}$

``` octave
W = rand(L_out, 1 + L_in) * 2 * epsilon_init - epsilon_init;
```

Reasonable default: 1 hidden layer

if > 1 hidden layer, keep the same amount of units in every layer (usually the more the better)

\# units: [ # inputs , 3 x # inputs ]

Training a neural network

1.   Randomly initialize weights
2.   Implement forward propagation to get $h_\Theta(x^{(i)})$
3.   Implement code to compute cost function $J(\Theta)$
4.   Implement backprop to compute partial derivatives $ \dfrac{\partial J(\Theta)}{\partial \Theta_{jk}^{(l)}}  $
5.   Gradient checking
6.   Use optmization with backprop to try to minimize $ J(\Theta) $ (non convex, gradient descent could converge to local minima)

``` octave
for i = 1:m,
   Perform forward propagation and backpropagation using example (x(i),y(i))
   (Get activations a(l) and delta terms d(l) for l = 2,...,L
```

```

```

# What to try next

1.   Get more training examples: fixes high variance
2.   Try smaller sets of features: fixes high variance
3.   Try getting additional features: fixes high bias
4.   Try adding polynomial features: fixes high bias
5.   Try decreasing $\lambda$: fixes high bias
6.   Try increasing $\lambda$: fixes high variance

# ML diagnostic
1.   Can give guidance
2.   Can be time-consuming to implement and try, but can be worth
3.   Can rule out certain actions

# Evaluate our hypothesis
1.   Random split training set (70%) and test set (30%)
2.   Fit with training data, minimizing cost function
3.   Compute test set error
4.   Misclassification error

# Random split
1.   Training set (60%)
2.   Cross validation set (20%)
3.   Test set (20%)

# Procedure
1.   Fit training set
2.   Cross validate on validation set
3.   Estimate error on test set

$$ err(h_\Theta(x),y) =
\begin{matrix}
1 & \mbox{if } h_\Theta(x) \geq 0.5\
and\ y = 0\
or\ h_\Theta(x) < 0.5\ and\ y = 1\newline
0 & \mbox otherwise\end{matrix}$$

# Bias or Variance

1. High bias (underfit) : $J_{train}(\theta)\approx J_{cv}(\theta): high,\ \lambda: high$, simple NN, cheaper
2. Just right, $\lambda: intermediate$
3. High variance (overfit): $J_{train}(\theta): low\ \lt\lt\ J_{cv}(\theta): high,\ \lambda: small$, increase training set could help, large NN, expensive: icnrease $\lambda$

# Regularization

1. Create a list of lambdas (i.e. $\lambda\in\{0,0.01,0.02,0.04,0.08,0.16,0.32,0.64,1.28,2.56,5.12,10.24\}$);
2. Create a set of models with different degrees or any other variants.
3. Iterate through the $\lambda$ and for each $\lambda$ go through all the models to learn some $\Theta$.
4. Compute the cross validation error using the learned $\Theta$ (computed with $\lambda$) on the $J_{CV}(\Theta)$ without regularization or $\lambda=0$.
5. Select the best combo that produces the lowest error on the cross validation set.
6. Using the best combo $\Theta$ and $\lambda$, apply it on $J_{test}(\Theta)$ to see if it has a good generalization of the problem.

# ML system design

Spam classifier

* Collect lots of data ("honeypot" project)
* develop sophisticated features (from header)
* develop sophisticated features (from body)
* develop sophisticated algorithm to detect misspellings

# Error analysis
* implement simple algorithm quickly (one day) and test on your cross validation data
* plot learning curves to decide if more data, features are likely to help
* manual error analysis
 * categorize
 * identify candidates features to correct
* define a numerical evaluation: cross validation error

# Error Metrics for Skewed Classes

|           |   | Actual | class |
|-----------|---|--------|-------|
|           |   | 1      | 0     |
| Predicted | 1 | TP     | FP    |
| class     | 0 | FN     | TN    |

* A good classifier should have both a high precision and high recall on the cross validation set

* Do you identify the actual positive?
$$ \text{Recall}
= \frac{\text{true positives}}{\text{# of actual positive}}
= R = \frac{TP}{TP+FN} $$

* Are your positive predictions correct?
$$ \text{Precision}
= \frac{\text{true positives}}{\text{# of predicted positive}}
= P = \frac{TP}{TP+FP} $$

|                | Threshold  | Precision     | Recall       |
|-----------------------|-----|---------------|--------------|
| Baseline              | 0.5 | Intermediate  | Intermediate |
| Avoid false positives | 0.7 | High          | Low          |
| Avoid false negatives | 0.3 | Low           | High         |

$$ \text{Average}
= \frac{P+R}{2}
$$

<!---
$$ \text{average}
= \frac{P+R}{2}
= \frac{1}{2}\left(\frac{TP}{TP+FP} + \frac{TP}{TP+FN}\right)
= \frac{1}{2}\left(\frac{TP(TP+FN+TP+FP)}{(TP+FP)(TP+FN)}\right)
= \frac{1}{2}\left(\frac{TP(TP+FN+TP+FP)}{(TP+FP)(TP+FN)}\right)$$
-->

\begin{align}
F_1 \;\text{score}
&= 2\cdot\frac{P\cdot R}{P+R}
= 2\cdot
\frac{\frac{TP}{TP+FP}\frac{TP}{TP+FN}}
{\frac{TP}{TP+FP}+\frac{TP}{TP+FN}}
\\
&= 2\cdot
\frac{\frac{(TP)^{2}}{(TP+FP)(TP+FN)}}
{\frac{TP(TP+FN+TP+FP)}{(TP+FP)(TP+FN)}}
= \frac{2(TP)}{2(TP)+FN+FP}
\end{align}

$$ F_1 \;\text{score}
= 2\cdot\frac{P\cdot R}{P+R} $$

$$ \text{Accuracy}
= \frac{\text{true positives}+\text{true negatives}}
{\text{# of examples}}
= \frac{TP+TN}{TP+FP+TN+FN} $$

|Alg| Precision (P)| Recall (R)| Average| $F_1 \text{score}$ |
|---|------|------|------|-------|
| 1 | 0.5  | 0.4  | 0.45 | 0.444 |
| 2 | 0.7  | 0.1  | 0.4  | 0.175 |
| 3 | 0.02 | 1.0  | 0.51 | 0.039 |

```
#TP = 80; FP = 20;
#FN = 80; TN = 820;

TP = 85; FP = 890;
FN = 15; TN = 10;

# Actual: 1: 1% spam / 0: 99% not spam

# predict all spam (y=1)
#TP = 1; FP = 99;
#FN = 0; TN = 0;

# predict all not spam (y=0)
TP = 0; FP = 0;
FN = 1; TN = 99;

'''
The classifier achieves 99% accuracy on the training set because of
how skewed the classes are. We can expect that the cross-validation set
will be skewed in the same fashion,
so the classifier will have approximately the same accuracy.

Since 99% of the examples are y = 0,
always predicting 0 gives an accuracy of 99%.
Note, however, that this is not a good spam system,
as you will never catch any spam.
'''

print("Recall: ", recall(TP,FN))
n = TP + FP + TN + FN
print("Accuracy: ", accuracy(TP,TN,n))

def precision(TP,FP):
  return TP / (TP+FP)

def recall(TP,FN):
  return TP / (TP+FN)

def average(P,R):
  return (P+R)/2

def f1_score(P,R):
  return 2*P*R/(P+R)

def accuracy(TP,TN,n):
  return (TP+TN)/n

P = precision(TP,FP)
R = recall(TP,FN)
# Total examples
n = TP + FP + TN + FN

print("Precision: ", P)
print("Recall: ", R)
print("Average: ", average(P,R))
print("F1 Score: ", f1_score(P,R))
print("Accuracy: ", accuracy(TP,TN,n))
```

```
import numpy as np

P = [0.5, 0.7, 0.02]
R = [0.4, 0.1, 1.0]

P = np.array(P)
R = np.array(R)

def average(P,R):
  return (P+R)/2

def f1_score(P,R):
  return 2*P*R/(P+R)

print("Average: ", average(P,R))
print("F1 Score: ", f1_score(P,R))
```

    Average:  [0.45 0.4  0.51]
    F1 Score:  [0.44444444 0.175      0.03921569]

# Data for ML

* features contain sufficient information to predict, e.g. human expert can predict y when given only x
* low bias with many parameters could use large dataset, if few parameters it will underfit
* Threshold
 * Low = more y = 1 predictions.
 So more true positives and less false negatives
 Therefore higher recall
 * High = more y = 0 predictions.
 So less true positives and more false negatives.
 Therefore lower recall
* Using very large training set makes it unilikely for model to overfit
 * A sufficiently large training set will not be overfit, as the model cannot overfit some of the examples without doing poorly on the others.
* on skewed datases (much more positive or negative), accuracy is not a good metric, you should use F-score based on precision and recall
* if your model is underfitting, obtain more data is unlikely to help

# SVM: Support Vector Machine

Large margin classifier

It could be derived from an alternative interpretation of the logistic regression.

The hypothesis function of the Logistic Regression is:

$$ h_\theta(x) = \frac{1}{1+e^{-\theta^Tx}} $$

Let's simplify the notation with $ z = -\theta^Tx $

$$ h_\theta(z) = \frac{1}{1+e^{-z}} $$

Optimization convex: global minimum

```
import numpy as np
import matplotlib.pyplot as plt

plt.style.use(['dark_background'])
plt.rcParams.update({'font.size': 14})

z = np.arange(-5, 5., 0.1)

def LogReg_hypothesis(z):
  hypothesis = 1./(1 + np.exp(-z))
  return hypothesis

hypothesis = LogReg_hypothesis(z)
plt.scatter(z, hypothesis)

plt.legend(['$\\dfrac{1}{1+e^{-z}}$'])
plt.xlabel('$z = \\theta^{T}x $')
plt.ylabel('Hypothesis: $h_\\theta(x)$');

```

Its cost function
$$ \mathrm{Cost}(h_\theta(x),y) = -y \log(h_\theta(x)) - (1-y)\log(1-h_\theta(x)) $$

As we replace the simplified Logistic Regression hypothesis  function

$$ \mathrm{Cost}(z,y) = -y \log\frac{1}{1+e^{-z}} - (1-y)\log\left(1-\dfrac{1}{1+e^{-z}}\right) $$

Let's look to the extreme cases:
1. $ y=1 \implies \mathrm{cost}_1 = -\log\;\dfrac{1}{1+e^{-z}} $
2. $ y=0 \implies \mathrm{cost}_2 = -\log\;\left(1-\dfrac{1}{1+e^{-z}}\right)$

```
z = np.arange(-3, 3., 0.1)

def LogReg_cost_function(z, y):
  cost = - y*np.log(1./(1 + np.exp(-z)))-(1-y)*np.log(1- 1./(1 + np.exp(-z)))
  return cost

# plt.legend(['$\\text{cost}_1$, if $y=1$','$\\text{cost}_2$, if $y=0$'])
cost1 = LogReg_cost_function(z, y=1)
#cost_1 = -np.log(1./(1 + np.exp(-z)))
cost2 = LogReg_cost_function(z, y=0)
#cost_2 = -np.log(1- 1./(1 + np.exp(-z)))

plt.figure(figsize=(8,6))
plt.plot(z, cost1, 'bs',
         z, cost2, 'g^')
plt.legend(['$ y=1,\, cost_1 = -\log\;\\dfrac{1}{1+e^{-z}}$',
            '$ y=0,\; cost_2 = -\log\;\\left(1-\\dfrac{1}{1+e^{-z}}\\right)$'])
plt.xlabel('$z = \\theta^{T}x $')
plt.ylabel('Cost Function: $J(h_\\theta(x),y) $')
#plt.axis([0, 1, 0, 5])
plt.show()
```

If we consider the regularization terms

\begin{equation*}
\mathop{\mbox{min}}_\theta\
\frac{1}{m}\sum_{i=1}^{m}
\left[
  y^{(i)} \text{cost}_1(\theta^Tx^{(i)})
  + (1-y^{(i)}) \text{cost}_0(\theta^Tx^{(i)})
\right]
+\frac{\lambda}{2m} \sum_{j=1}^n \theta^2_j
\end{equation*}

```
import pandas as pd

z = np.arange(-3, 3., 0.1)

cost1 = np.zeros(len(z))
idx = np.where(z<1)
cost1[idx] = SVM_linear(z[idx],a=-1,b=1)

cost0 = np.zeros(len(z))
idx = np.where(z>-1)
cost0[idx] = SVM_linear(z[idx],a=1,b=1)

plt.plot(z,cost0,label='cost0')
plt.plot(z,cost1,label='cost1')
plt.legend();

pd.DataFrame({'z': z,
              'cost1': cost1,
              'cost2': cost2}).round(2).head()
```

We could rewrite this as:

\begin{equation*}
\mathop{\mbox{min}}_\theta\
C \sum_{i=1}^{m}
\left[
  y^{(i)} \text{cost}_1(\theta^Tx^{(i)})
  + (1-y^{(i)}) \text{cost}_0(\theta^Tx^{(i)})
\right]
+\frac{1}{2} \sum_{j=1}^n \theta^2_j
\end{equation*}

$J = y^{(i)} \text{cost}_1(\theta^Tx^{(i)}) + (1-y^{(i)}) \text{cost}_0(\theta^Tx^{(i)}) $

* if $y^{(i)}=0 \implies J = \text{cost}_0(\theta^Tx^{(i)})$
 * if $\theta^Tx^{(i)}\le -1 \implies \text{cost}_0 = 0$
* if $y^{(i)}=1 \implies J = \text{cost}_1(\theta^Tx^{(i)})$
 * if $\theta^Tx^{(i)}\ge 1 \implies \text{cost}_1 = 0$

Therefore $J=0$
* if $y^{(i)}=0$ and $\theta^Tx^{(i)}\le -1$ or
* if $y^{(i)}=1$ and $\theta^Tx^{(i)}\ge 1$

if the regularization parameter $C$ is very large, $C=100.000$, so is highly sensible to outliers

$$ C \approx \frac{1}{\lambda} $$

```
z = np.arange(-3, 3., 0.1)

def SVM_linear(z,a,b):
  return a*z + b

def SVM_cost0(z):
  cost0 = np.zeros(len(z))
  idx = np.where(z>-1)
  cost0[idx] = SVM_linear(z[idx],a=1,b=1)
  return cost0

def SVM_cost1(z):
  cost1 = np.zeros(len(z))
  idx = np.where(z<1)
  cost1[idx] = SVM_linear(z[idx],a=-1,b=1)
  return cost1

def SVM_cost_function(z, y):
  cost = y*SVM_cost1(z) + (1-y)*SVM_cost0(z)
  return cost

cost1 = SVM_cost_function(z, y=1)
cost2 = SVM_cost_function(z, y=0)

plt.figure(figsize=(8,6))
plt.plot(z, cost1, 'bs',
         z, cost2, 'g^')
plt.legend(['$ y=1,\, cost_1 $',
            '$ y=0,\; cost_0 $'])
plt.xlabel('$z = \\theta^{T}x $')
plt.ylabel('Cost Function: $J(z,y) $')
plt.show()
```

```
x1 = np.arange(-1, 6., 0.1)
x2 = np.arange(-1, 6., 0.1)
theta0 = -3
theta1 = 1
theta2 = 0
y = theta0 + theta1*x1 + theta2*x2
predict = np.zeros(len(y))
idx = np.where(y>0)
predict[idx] = 1
predict
```

    array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1., 1.])

## Kernels

Non-linear decision boundary

Gaussian

Given $x$, compute new feature $f_i$ depending on proximity to landmarks $l$ (new features)

\begin{equation*}
f_1 =\text{similarity}(x,l^{(1)})
= \exp\left(-\frac{\left\Vert x-l^{(1)} \right\Vert^2}{2\sigma^2}\right)
\end{equation*}

$sim(x,l^{(1)}) \in [0,1]$

If $ x = l^{(1)} $:
\begin{equation*}
f_1
= \exp\left(-\frac{0^2}{2\sigma^2}\right)
= 1
\end{equation*}

If $ x \approx l^{(1)} $:
\begin{equation*}
f_1
\approx \exp\left(-\frac{0^2}{2\sigma^2}\right)
\approx 1
\end{equation*}

If $x$ far from $l^{(1)}$:
\begin{equation*}
f_1
\approx \exp\left(-\frac{\left(\text{large number}\right)^2}{2\sigma^2}\right)
\approx 0
\end{equation*}

* C: regularization parameter (large: all data must fit, including outlier, overfit)
* $\sigma$ bandwith how fast the similarity metric decreases (to 0)

* Large C
 * lower bias
 * higher variance (overfit)
 * ~ small $\lambda$
* Small C
 * higher bias (underfit)
 * lower variance
 * ~ large $\lambda$

* Large $\sigma^2$
 * features $f_i$ vary more smoothly
 * higher bias
 * lower variance

* Small $\sigma^2$
 * features $f_i$ vary less smoothly
 * lower bias
 * higher variance

* Overfit
 * Decrease C
 * Increase $\sigma^2$

* Underfit
 * Increase C
 * Decrease $\sigma^2$

Options for kernels
1. No kernels (Linear Kernel)
 * n (feature) large, m (train set) small

2. Gaussian Kernel
 * need to choose $\sigma^2$
 * n small and/or m large
 * perform feature scaling

Other kernels
* polynomial : $(x^T l + \text{constant})^\text{degree}$
* string, chi-square, histogram, intersection

Multi-class classification
* packages
* one vs all method
 * $K$ classes $ \implies K$ classifiers

```
l_1 = 5
x1 = 3

x = np.arange(0,10,0.1)
sigma = 1
f_1 = np.exp(-(x-l_1)**2/(2*sigma**2))
plt.plot(x,f_1);
```

```
sigma = 2
f_1 = np.exp(-(x-l_1)**2/(2*sigma**2))
plt.plot(x,f_1);
```

# Logistic regression vs SVMs
* $n$: number of features
* $m$: number of training examples

If $n \gg m $ ($n=10.000$, $m=10\cdot\cdot\cdot1.000$)
* Logistic regression
* SVM without a kernel

If $n$ is small, $m$ is intermediate ($n=1\cdot\cdot\cdot1.000$, $m=10\cdot\cdot\cdot10.000$)
 * SVM with Gaussian kernel

If $n \ll m $ ($n=1\cdot\cdot\cdot1.000$, $m=50.000+$)
* Create/add more features
 * Logistic regression
 * SVM without a kernel

Neural network likely to work well for most of these settings, but may be slower to train

Logistic regression and SVM without a kernel
* provides a linear straight line decision boundary

$n=10$ features and $m=5000$ examples
* logistic regression classifier with gradient descent
 * training set: underfit (reduced performance on training or cross-validation sets)

* create/ add new polynomial features
* logistic regression cost function is convex, so gradient descent will always find the global minimum

# Spam

https://spamassassin.apache.org/old/publiccorpus/

```

```

```
# TODO
import sys

z = np.arange(-3, 3., 0.1)

def SVM_linear(z,a,b):
  return a*z - b

def SVM_cost_function(z, y):
  cost = np.zeros(len(z))
  b = 0.5
  if y == 1:
    cost1 = [SVM_linear(i,a=-1,b) if z<1 for i in z]
  elif y == 0:
    cost0 = [SVM_linear(i,a=1,b) if z>-1 for i in z]
  else:
    sys.exit('ERROR: Not programmed such y. Please enter 0 or 1')

  return cost

cost1 = np.zeros(len(z))

cost1[z<1] = [SVM_linear(i,a=-1,b) if z>-1 for i in z]
```


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## White Noise

### Definition

[White noise](https://en.wikipedia.org/wiki/White_noise) is a wide-sense stationary (WSS) random signal with constant power spectral density (PSD)

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = N_0
\end{equation}

where $N_0$ denotes the power per frequency. White noise draws its name from the analogy to white light. It refers typically to an idealized model of a random signal, e.g. emerging from measurement noise. The auto-correlation function (ACF) of white noise can be derived by inverse discrete-time Fourier transformation (DTFT) of the PSD

\begin{equation}
\varphi_{xx}[\kappa] = \mathcal{F}_*^{-1} \{ N_0 \} = N_0 \cdot \delta[\kappa]
\end{equation}

This result implies that white noise has to be a zero-mean random process. It can be concluded from the ACF that two neighboring samples $k$ and $k+1$ are uncorrelated. Hence they show no dependencies in the statistical sense. Although this is often assumed, the probability density function (PDF) of white noise is not necessarily given by the normal distribution. In general, it is required to additionally state the amplitude distribution when denoting a signal as white noise.

### Example - Amplifier Noise

Additive white Gaussian noise (AWGN) is often used as a model for amplifier noise. In order to evaluate if this holds for a typical audio amplifier, the noise captured from a microphone preamplifier at full amplification with open connectors is analyzed statistically. For the remainder, a function is defined to estimate and plot the PDF and ACF of a given random signal.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats
%matplotlib inline

def estimate_plot_pdf_acf(x, nbins=50, acf_range=30):

    # compute and truncate ACF
    acf = 1/len(x) * np.correlate(x, x, mode='full')
    acf = acf[len(x)-acf_range-1:len(x)+acf_range-1]
    kappa = np.arange(-acf_range, acf_range)

    # plot PSD
    plt.figure(figsize = (10, 6))
    plt.subplot(121)
    plt.hist(x, nbins, density=True)
    plt.title('Estimated PDF')
    plt.xlabel(r'$\theta$')
    plt.ylabel(r'$\hat{p}_x(\theta)$')
    plt.grid()

    # plot ACF
    plt.subplot(122)
    plt.stem(kappa, acf)
    plt.title('Estimated ACF')
    plt.ylabel(r'$\hat{\varphi}_{xx}[\kappa]$')
    plt.xlabel(r'$\kappa$')
    plt.axis([-acf_range, acf_range, 1.1*min(acf), 1.1*max(acf)]);
    plt.grid()
```

Now the pre-captured noise is loaded and analyzed

```python
noise = np.load('../data/amplifier_noise.npz')['noise']
estimate_plot_pdf_acf(noise, nbins=100, acf_range=150)
```

Inspecting the PDF reveals that it fits quite well to a [normal distribution](important_distributions.ipynb#Normal-Distribution). The ACF consists of a pronounced peak. from which can be concluded that the samples are approximately uncorrelated. Hence, the amplifier noise can be modeled reasonably well as additive white Gaussian noise. In oder to estimate the parameters of the normal distribution, the captured samples are fitted to a normal distribution

```python
mean, sigma = stats.norm.fit(noise)
print('Mean: {0:1.3e} \nVariance: {1:1.3e}'.format(mean, sigma**2))
```

    Mean: -1.537e-05
    Variance: 1.140e-07

**Excercise**

* What relative level does the amplifier noise have when the maximum amplitude of the amplifier is assumed to be $\pm 1$?

Solution: The average power of a mean-free random signal is given by is variance, here $\sigma_\text{noise}^2$. Due to the very low mean in comparison to the maximum amplitude, the noise can be assumed to be mean-free. Hence, the relative level of the noise is then given as $10*\log_{10}\left( \frac{\sigma_\text{noise}^2}{1} \right)$. Numerical evaluation yields

```python
print('Level of amplifier noise: {:2.2f} dB'.format(10*np.log10(sigma**2/1)))
```

    Level of amplifier noise: -69.43 dB

### Example - Generation of White Noise with Different Amplitude Distributions

Toolboxes for numerical mathematics like `Numpy` or `scipy.stats` provide functions to draw uncorrelated random samples with a given amplitude distribution.

**Uniformly distributed white noise**

For samples drawn from a zero-mean random process with uniform amplitude distribution, the PDF and ACF are estimated as

```python
np.random.seed(3)
estimate_plot_pdf_acf(np.random.uniform(size=10000)-1/2)
```

Lets listen to uniformly distributed white noise

```python
from scipy.io import wavfile
fs = 44100

x = np.random.uniform(size=5*fs)-1/2
wavfile.write('uniform_white_noise.wav', fs, np.int16(x*32768))
```

<audio src="./uniform_white_noise.wav" controls>Your browser does not support the audio element.</audio>[./uniform_white_noise.wav](./uniform_white_noise.wav)

**Laplace distributed white noise**

For samples drawn from a zero-mean random process with with Laplace amplitude distribution, the PDF and ACF are estimated as

```python
estimate_plot_pdf_acf(np.random.laplace(size=10000, loc=0, scale=1/np.sqrt(2)))
```

**Exercise**

* Do both random processes represent white noise?
* Estimate the power spectral density $N_0$ of both examples.
* How does the ACF change if you lower the length `size` of the random signal. Why?

Solution: Both processes represent white noise since the ACF can be approximated reasonably well as Dirac impulse $\delta[\kappa]$. The weight of the Dirac impulse is equal to $N_0$. In case of the uniformly distributed white noise $N_0 \approx \frac{1}{12}$, in case of the Laplace distributed white noise $N_0 \approx 1$. Decreasing the length `size` of the signal increases the statistical uncertainties in the estimate of the ACF.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.