# T자 모양 단면의 도심
Centroid of a T shaped Section

```python
# 그래프, 수학 기능 추가
# Add graph and math features
import pylab as py
import numpy as np
import numpy.linalg as nl
# 기호 연산 기능 추가
# Add symbolic operation capability
import sympy as sy

```

참고문헌 : 예제 5.2, Pytel 외 저, 이주성 외 역, 재료역학, 2판, 한티미디어, 2013.
Ref: Example 5.2, Pytel, Kiusalaas, Sharma, Mechanics of Materials, 2nd Ed., Cengage Learning, 2013.

다음과 같은 T자 모양 단면의 도심을 구해 보자.

Let's try to find the centroid of the following T shaped section.

윗 변의 폭 $w=150mm$
Width of the upper side $w=150mm$

```python
w_mm = 150

```

아랫변의 높이 $h=200mm$
Height of the lower side $h=200mm$

```python
h_mm = 200

```

두께 $t=20mm$
Thickness $t=20mm$

```python
t_mm = 20

```

## 도심의 정의
Definition of a Centroid

$$
C_y=\frac{\int yS_x(y) dy}{A}=\frac{\int yS_x(y) dy}{\int S_x(y) dy}
$$

ref : https://en.wikipedia.org/wiki/Centroid

여기서 $S_x(y)$는 다음과 같다. (T 단면의 아래 끝에서 $y=0$)

Here, $S_x(y)$ is as follows. ($y=0$ at the lower end of T section)

$$
S_x(y) =
\begin{cases}
t, & 0 \leq y < h \\
w, & h \leq y < h + t \\
0, & otherwise
\end{cases}
$$

Python 언어로는 다음과 같이 구현할 수 있다.
We can implement in python as follows.

```python
def sx(y_mm):
    if 0 <= y_mm < h_mm :
        result = t_mm
    elif h_mm <= y_mm < (h_mm + t_mm):
        result = w_mm
    else:
        result = 0

    return result

```

이 함수의 그래프를 그려 보자
Let's plot this.

```python
y_mm_array = py.arange(0, h_mm + t_mm + 0.5, 1)
sx_mm_array = py.array([sx(y_mm) for y_mm in y_mm_array])

py.plot(sx_mm_array * 0.5, y_mm_array)
py.plot(sx_mm_array * (-0.5), y_mm_array)

py.axis('equal')
py.grid(True)

py.xlabel('x(mm)')
py.xlabel('y(mm)')

```

## 정적분 계산
Numerical Integration

0차 적분 함수를 이용해 보자
Let's use 0'th order numerical integration function.

```python
def num_int_0(f, xi, xe, delta_x):
    x_array = py.arange(xi, xe+delta_x*0.1, delta_x)

    integration_result = 0.0

    for x_i in x_array:
        y_i = f(x_i)
        area_i = delta_x * y_i
        integration_result += area_i

    return integration_result

```

### 단면적
Area of the section

```python
A_mm2 = num_int_0(sx, 0, h_mm + t_mm, 1)

```

```python
A_mm2

```

확인해 보자.
Let's verify.

```python
h_mm * t_mm + w_mm * t_mm

```

아래와 같이 지정해 두면 T 자 단면적 결과가 맞는지 확인할 수 있다.

We can designate as follows to assert T shape section area.

```python
assert 1e-6 > abs((h_mm * t_mm + w_mm * t_mm) - A_mm2)

```

### 도심
Centroid

```python
def ySx(y_mm):
    return y_mm * sx(y_mm)

```

```python
numerator_mm3 = num_int_0(ySx, 0, h_mm + t_mm, 1)

```

```python
cy_mm = numerator_mm3 / A_mm2

```

```python
cy_mm

```

역시 확인해 보자.
Again, let's verify.

```python
cy_exact_mm = ((h_mm * t_mm) * (h_mm * 0.5) + (w_mm * t_mm) * (h_mm + 0.5 * t_mm)) / (h_mm * t_mm + w_mm * t_mm)

```

```python
cy_exact_mm

```

```python
cy_mm - cy_exact_mm

```

어떻게 하면 위 오차를 줄일 수 있을 것인가?
How can we make the error above smaller?

```python
error = (cy_mm - cy_exact_mm)

```

```python
try :
    assert (1e-6 > abs(error)), "Error too large"
except AssertionError as e:
    print(e)

```

## Final Bell
마지막 종

```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");

```

```python

```


# Upper envelope

This notebook shows how to use the **upperenvelope** module from the **consav** package.

# Model

Consider a **standard consumption-saving** model

\begin{align}
v_{t}(m_{t})&=\max_{c_{t}}\frac{c_{t}^{1-\rho}}{1-\rho}+\beta v_{t+1}(m_{t+1})
\end{align}

where

\begin{align}
a_{t}	&=m_{t}-c_{t} \\
m_{t+1}	&=Ra_{t}+y \\
\end{align}

The **Euler equation** is

\begin{align}
c_{t}^{-\rho} &=\beta Rc_{t+1}^{-\rho}
\end{align}

Assume that the **t+1 consumption and value functions** are given by

\begin{align}
c_{t+1}(m_{t}) &= \sqrt{m_{t}}-\eta_{c} \cdot 1\{m_{t}\geq\underline{m}\} \\
v_{t+1}(m_{t}) &= \sqrt{m_{t}}+\eta_{v}\sqrt{m_{t}-\underline{m}} \cdot 1\{m_{t}\geq\underline{m}\}
\end{align}

This **notebook** shows how to find the **t consumption and value function** using an **upper envelope** code despite the **kink** in the next-period value function.

# Algorithm

1. Specify an increasing grid of $m_t$ indexed by $j$, such as {${m_1,m_2,...,m_{\#_m}}$}

2. Specify an increasing grid of $a_t $ indexed by $i$, such as {${a^1,a^2,...,a^{\#_a}}$}

3. For each $i$ compute (using linear interpolation):

  a. Post-decision value function: $w^i = \beta \breve{v}_{t+1}(Ra^i+y)$

  b. Post-decision marginal value of cash: $q^i = \beta R\breve{c}_{t+1}(Ra^i+y)^{-\rho}$

  c. Consumption: $c_i = (q^i)^{-1/\rho}$

  d. Cash-on-hand: $m^i = a^i + c^i$

4. For each $j$:

  a. Constraint: If $m_j < m^1$ then set $c_j = m_j$

  b. Find best segment: If $m_j \geq m^1$ then set $c_j =c_j^{i^{\star}(j)} $ where

  $$
  \begin{align}
  c_j^i=c_j^i+\frac{c^{i+1}-c^i}{m^{i+1}-m^i}(m_j-m^i)
  \end{align}
  $$

  and

  $$
  \begin{align}
  i^{\star}(j)=\arg\max_{i\in\{1,\dots\#_{A}-1\}}\frac{(c_{j}^{i})^{1-\rho}}{1-\rho}+\beta w_{j}^{i} \\
  \end{align}
  $$

  subject to

  $$
  \begin{align}
  m_{j} &\in [m^{i},m^{i+1}] \\
  a_{j}^{i} &= m_{j}-c_{j}^{i} \\
  w_{j}^{i} &= w^{i}+\frac{w^{i+1}-w^{i}}{a^{i+1}-a^{i}}(a_{j}^{i}-a^{i})
  \end{align}
  $$

# Setup

```python
import numpy as np
from numba import njit

import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
prop_cycle = plt.rcParams["axes.prop_cycle"]
colors = prop_cycle.by_key()["color"]
import ipywidgets as widgets
```

Choose parameters and create grids:

```python
def setup():

    par = dict()

    # a. model parameters
    par['beta'] = 0.96
    par['rho'] = 2
    par['R'] = 1.02
    par['y'] = 1

    # b. cash-on-hand (exogenous grid)
    par['Nm'] = 10000
    par['m_max'] = 10

    # c. end-of-period assets (exogenous grid)
    par['Na'] = 1000
    par['a_max'] = 10

    # d. next-period consumption and value function
    par['eta_v'] = 0.5
    par['eta_c'] = 0.5
    par['x_ubar'] = 5

    return par

def create_grids(par):

    par['grid_a'] = np.linspace(0,par['a_max'],par['Na'])
    par['grid_m'] = np.linspace(1e-8,par['m_max'],par['Nm'])

    return par

par = setup()
par = create_grids(par)
```

# Next-period functions

Calculate the next-period consumption and value functions:

```python
sol = dict()

# a. consumption function
sol['c_next'] = np.sqrt(par['grid_m']) - par['eta_c']*(par['grid_m'] >= par['x_ubar']);

# b. value function
sol['v_next'] = np.sqrt(par['grid_m']) + par['eta_v']*np.sqrt(np.fmax(par['grid_m']-par['x_ubar'],0))*(par['grid_m'] >= par['x_ubar'])
```

## Figures

Plot them to see the jump in consumption and the kink in the value function.

```python
# a. next-period consumption function
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(par['grid_m'],sol['c_next'],'o',MarkerSize=0.5)

ax.set_title('next-period consumption function')
ax.set_xlabel('$m_t$')
ax.set_ylabel('$c_t$')

# b. next-period value function
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(par['grid_m'],sol['v_next'],'o',MarkerSize=0.5)

ax.set_title('next-period value function')
ax.set_xlabel('$m_t$')
ax.set_ylabel('$v_t$');
```

# EGM

```python
from consav import linear_interp # linear interpolation
```

Apply the EGM algorithm.

```python
@njit
def u(c,rho):
    return c**(1-2)/(1-2)

def marg_u(c,par):
    return c**(-par['rho'])

def inv_marg_u(u,par):
    return u**(-1.0/par['rho'])

def EGM(par,sol):

    # a. next-period cash-on-hand
    m_plus = par['R']*par['grid_a'] + par['y']

    # b. post-decision value function
    sol['w_vec'] = np.empty(m_plus.size)
    linear_interp.interp_1d_vec(par['grid_m'],sol['v_next'],m_plus,sol['w_vec'])

    # c. post-decision marginal value of cash
    c_next_interp = np.empty(m_plus.size)
    linear_interp.interp_1d_vec(par['grid_m'],sol['c_next'],m_plus,c_next_interp)
    q = par['beta']*par['R']*marg_u(c_next_interp,par)

    # d. EGM
    sol['c_vec'] = inv_marg_u(q,par)
    sol['m_vec'] = par['grid_a'] + sol['c_vec']

    return sol

sol = EGM(par,sol)
```

## Figures

Plot the result of the EGM algorithm to see that the its does not define a consumption function.

```python
# a. raw consumption function
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(sol['m_vec'],sol['c_vec'],'o',MarkerSize=0.5)

ax.set_title('raw consumption points')
ax.set_xlabel('$m_t$')
ax.set_ylabel('$c_t$')

# b. raw value function
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(sol['m_vec'],sol['w_vec'],'o',MarkerSize=0.5)

ax.set_title('raw value function points')
ax.set_xlabel('$m_t$')
ax.set_ylabel('$w_t$');
```

# Upper envelope

```python
from consav import upperenvelope

# a. create
myupperenvelope = upperenvelope.create(u) # where is the utility function

# b. apply
c_ast_vec = np.empty(par['grid_m'].size) # output
v_ast_vec = np.empty(par['grid_m'].size) # output
myupperenvelope(par['grid_a'],sol['m_vec'],sol['c_vec'],sol['w_vec'],par['grid_m'],c_ast_vec,v_ast_vec,par['rho'])
```

## Figures

```python
# a. consumption function
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(par['grid_m'],c_ast_vec,'o',MarkerSize=0.5)

ax.set_title('consumption function')
ax.set_xlabel('$m_t$')
ax.set_ylabel('$c_t$')

# b. value function
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(par['grid_m'],v_ast_vec,'o',MarkerSize=0.5)

ax.set_title('value function')
ax.set_xlabel('$m_t$')
ax.set_ylabel('$v_t$')

ax.set_ylim((-5,5));
```


# Modelo del rendimiento de una cuenta de ahorro

> **¿Tiene el dinero el mismo valor a lo largo del tiempo?** La respuesta es *no*. Todos lo hemos vivido.

> Dos situaciones básicas:
1. Inflación: ¿Cuánto dinero necesitabas para comprar unas papas y un refresco hace 10 años? ¿Cuánto necesitas hoy?
2. Interés: no es lo mismo tener $\$10000$ MXN disponibles hoy a recibir $\$10000$ MXN en un año, pues los primeros pueden ser invertidos en un negocio o una cuenta bancaria para generar *interés*. Por lo tanto los $\$10000$ MXN disponibles hoy valen más que los $\$10000$ MXN que se recibirán en un año.

Referencia:
- Vidaurri Aguirre, Héctor Manuel. *Ingeniería económica básica*, ISBN: 978-607-519-017-4. (Disponible en biblioteca)

Referencias:
- http://www.sympy.org
- http://matplotlib.org
- http://www.numpy.org
- http://ipywidgets.readthedocs.io/en/latest/index.html

___
## Interés
Nos centraremos en como cambia el valor del dinero en el tiempo debido al **interés**. Existen dos tipos:

### Capitalización por interés simple
Este tipo de interés se calcula única y exclusivamente sobre la cantidad original que se invirtió. Como consecuencia, el interés generado no forma parte del dinero que se invierte, es decir, los intereses no ganan intereses.

Suponga que se tiene un capital inicial $C_0$ y se invierte a un plazo de $k$ periodos (pueden ser meses, trimestres, semestres, años...) a una tasa de **interés simple** por periodo $i$. Al final del primer periodo, el capital $C_1$ que se obtiene es:

$$C_1=C_0+iC_0=C_0(1+i).$$

De la misma manera, como el interés solo se calcula sobre el capital inicial, al final del segundo periodo, el capital $C_2$ que se obtiene es:

$$C_2=C_1+iC_0=C_0+iC_0+iC_0=C_0(1+2i).$$

Así, al final del $k-$ésimo periodo, el capital $C_k$ que se obtiene es:

$$C_k=C_{k-1}+iC_0=C_0+kiC_0=C_0(1+ki).$$

> **Ejemplo.** Suponga que se tiene un capital de \$10000 MXN, el cual se pone en un fondo de inversión que paga una tasa de interés simple del 0.8% mensual.

> Si se tiene una meta de ahorro de \$11000 MXN sin inversiones adicionales, ¿cuántos meses se debería dejar invertido el dinero?

```python
# Librería para cálculo numérico
import numpy as np
```

```python
# Valores dados en el enunciado
C0 = 10000
Ck = 11000
i = 0.008
```

```python
# Despejamos k tal que C_k=meta
k = np.ceil((Ck / C0 - 1) / i)  # Función techo (ceil): encuentra el entero superior más cercano
k
```

    13.0

```python
type(k)
```

    numpy.float64

```python
type(k.astype(int))
```

    numpy.int64

```python
# Imprimimos respuesta en pantalla
print(f"Para llegar a la meta de {Ck} con un capital inicial de {C0} a una tasa de {i} mensual "
      f"se necesitan {k} meses. El dinero al finalizar los {k} meses es: {C0 * (1 + k * i)}.")
```

    Para llegar a la meta de 11000 con un capital inicial de 10000 a una tasa de 0.008 mensual se necesitan 13.0 meses. El dinero al finalizar los 13.0 meses es: 11040.000000000002.

> **Actividad.**
> - ¿Qué pasa si el interés no es del 0.8% mensual sino del 1% mensual?
> - ¿Qué pasa si la meta no son $\$11000$ MXN si no $\$12000$ MXN?

```python
# Solución

```

> Una gráfica que nos permite ilustrar la situación anterior se puede realizar de la siguiente manera.

```python
# Librerías para gráficos
import matplotlib.pyplot as plt
# Para que se muestren las gráficas en la misma ventana
%matplotlib inline
# Librería para widgets de python interactivo
from ipywidgets import *
```

Explicar la siguiente función

```python
# Función de gráficos de interés simple
def interes_simple(C_0, meta, i):
    # Despejamos k
    k = np.ceil((meta / C_0 - 1) / i)  # Notar el uso de la función ceil
    k = k.astype(int)  # Conversión a entero
    C_k = C_0 * (1 + k * i)  # Cálculo del capital al final del periodo k
    C_k = round(C_k, 2)  # Redondeo con dos cifras decimales
    # Vector de periodos
    kk = np.linspace(0, k, k + 1)
    # Vector de capitales por periodo
    CC = C_0 * (1 + kk * i)
    # Gráfico
    plt.figure(num=1)  # Figura 1
    plt.clf()  # Borrar lo que contenga la figura
    plt.plot(kk, CC,'*',linewidth=3)  # Se grafica la evolución de los capitales
    plt.plot(kk, meta * np.ones(k + 1), '--k')  # Se grafica la meta
    plt.xlabel('k')  # Etiqueta eje x
    plt.ylabel('C_k')  # Etiqueta eje y
    plt.grid()  # Malla en la gráfica
    plt.show()  # Mostrar la figura
    print("El número de periodos que se debe dejar invertido el dinero para llegar a la meta de ", meta," es ", k,
          ". Al final del periodo ", k,", el capital es ", C_k, ".", sep="")
```

```python
interact_manual(interes_simple,
                C_0=fixed(10000),
                meta=(10000, 20000, 100),
                i=fixed(0.008))
```

    interactive(children=(IntSlider(value=15000, description='meta', max=20000, min=10000, step=100), Button(descr…

    <function __main__.interes_simple(C_0, meta, i)>

Como se esperaba, el capital en el $k-$ésimo periodo $C_k=C_0(1+ki)$ crece linealmente con $k$.

### Capitalización por interés compuesto
El capital que genera el interés simple permanece constante todo el tiempo de duración de la inversión. En cambio, el que produce el interés compuesto en un periodo se convierte en capital en el siguiente periodo. Esto es, el interés generado al final de un periodo se reinvierte para el siguiente periodo para también producir interés.

Suponga que se tiene un capital inicial $C_0$, y se va a ceder el uso de este capital por un periodo de tiempo determinado a una tasa de interés $i$. El capital que se obtiene al final del primer periodo $C_1$ se puede calcular por

$$C_1=C_0(1+i).$$

Si la anterior suma se vuelve a ceder a la misma tasa de interés, al final del periodo dos el capital $C_2$ es

$$C_2=C_1(1+i)=C_0(1+i)^2.$$

Si se repite el anterior proceso $k$ veces, el capital al final del $k-$ésimo periodo $C_k$ es

$$C_k=C_{k-1}(1+i)=C_0(1+i)^k.$$

**Referencia**:
- https://es.wikipedia.org/wiki/Inter%C3%A9s_compuesto.

> **Ejemplo.** Suponga que se tiene un capital de \$10000 MXN, el cual se pone en un fondo de inversión que paga una tasa de interés del 0.8% mensual.

> Si se tiene una meta de ahorro de \$11000 MXN sin inversiones adicionales, ¿cuántos meses se debería dejar invertido el dinero?

> Muestre una gráfica que ilustre la situación.

```python
def interes_compuesto(C_0, meta, i):
    # Despejamos k
    k = np.ceil(np.log(meta / C_0) / np.log(1 + i))
    k = k.astype(int)
    C_k = C_0 * (1 + i)**k  # Cálculo del capital al final del periodo k
    C_k = round(C_k, 2)  # Redondeo con dos cifras decimales
    # Vector de periodos
    kk = np.linspace(0, k, k + 1)
    # Vector de capitales por periodo
    CC = C_0 * (1 + i)**kk
    # Gráfico
    plt.figure(num=1)  # Figura 1
    plt.clf()  # Borrar lo que contenga la figura
    plt.plot(kk, CC, '*', linewidth=3)  # Se grafica la evolución de los capitales
    plt.plot(kk, meta * np.ones(k + 1), '--k')  # Se grafica la meta
    plt.xlabel('k')  # Etiqueta eje x
    plt.ylabel('C_k')  # Etiqueta eje y
    plt.grid()  # Malla en la gráfica
    plt.show()  # Mostrar la figura
    print("El número de periodos que se debe dejar invertido el dinero para llegar a la meta de ", meta," es ", k,
          ". Al final del periodo ", k,", el capital es ", C_k, ".", sep="")
```

```python
interact_manual(interes_compuesto,
                C_0=fixed(10000),
                meta=(10000, 20000, 100),
                i=fixed(0.008))
```

    interactive(children=(IntSlider(value=15000, description='meta', max=20000, min=10000, step=100), Button(descr…

    <function __main__.interes_compuesto(C_0, meta, i)>

El capital en el $k-$ésimo periodo $C_k=C_0(1+i)^k$ crece de manera exponencial con $k$.

> **Actividad.**
> - Modificar el código anterior para dejar fija la meta de ahorro y variar la tasa de interés compuesta.

```python

```

### Capitalización continua de intereses
La capitalización continua se considera un tipo de capitalización compuesta, en la que a cada instante de tiempo $t$ se se capitalizan los intereses. Es decir, la frecuencia de capitalización es infinita (o, equivalentemente, el periodo de capitalización tiende a cero).

Suponga que se tiene un capital inicial $C_0$, y que el capital acumulado en el tiempo $t$ es $C(t)$. Queremos saber cuanto será el capital pasado un periodo de tiempo $\Delta t$, dado que la tasa de interés efectiva para este periodo de tiempo es $i$. De acuerdo a lo anterior tenemos

$$C(t+\Delta t)=C(t)(1+i)=C(t)(1+r\Delta t),$$

donde $r=\frac{i}{\Delta t}$ es la tasa de interés instantánea. Manipulando la anterior expresión, obtenemos

$$\frac{C(t+\Delta t)-C(t)}{\Delta t}=r\; C(t).$$

Haciendo $\Delta t\to 0$, obtenemos la siguiente ecuación diferencial

$$\frac{d C(t)}{dt}=r\; C(t),$$

sujeta a la condición inicial (monto o capital inicial) $C(0)=C_0$.

La anterior, es una ecuación diferencial lineal de primer orden, para la cual se puede calcular la *solución analítica*.

```python
# Librería de cálculo simbólico
import sympy as sym
# Para imprimir en formato TeX
sym.init_printing(use_latex='mathjax')
```

```python
# Símbolos t(para el tiempo) y r(para el interés instantáneo)
sym.var("t r")
# Función de capital
C = sym.Function('C')
```

```python
# Ecuación diferencial
eq = sym.Eq(sym.Derivative(C(t), t) - r * C(t), 0)
# Mostrar ecuación
display(eq)
# Resolver
sym.dsolve(eq, C(t))
```

$\displaystyle - r C{\left(t \right)} + \frac{d}{d t} C{\left(t \right)} = 0$

$\displaystyle C{\left(t \right)} = C_{1} e^{r t}$

con $C_1=C_0$.

La equivalencia entre la tasa de interés compuesta $i$ y la tasa de interés instantánea $r$ viene dada por

$$e^r=1+i.$$

___
¿Cómo podemos calcular la *solución numérica*?

> **Ejemplo.** Suponga que se tiene un capital de \$10000 MXN, el cual se pone en un fondo de inversión que paga una tasa de interés del 0.8% mensual.

> Si se tiene una meta de ahorro de \$11000 MXN sin inversiones adicionales, ¿cuánto tiempo se debe dejar invertido el dinero?

> Muestre una gráfica que ilustre la situación.

```python
# Librerías para integración numérica
from scipy.integrate import odeint
```

```python
help(odeint)
```

    Help on function odeint in module scipy.integrate.odepack:

    odeint(func, y0, t, args=(), Dfun=None, col_deriv=0, full_output=0, ml=None, mu=None, rtol=None, atol=None, tcrit=None, h0=0.0, hmax=0.0, hmin=0.0, ixpr=0, mxstep=0, mxhnil=0, mxordn=12, mxords=5, printmessg=0, tfirst=False)
        Integrate a system of ordinary differential equations.

        .. note:: For new code, use `scipy.integrate.solve_ivp` to solve a
                  differential equation.

        Solve a system of ordinary differential equations using lsoda from the
        FORTRAN library odepack.

        Solves the initial value problem for stiff or non-stiff systems
        of first order ode-s::

            dy/dt = func(y, t, ...)  [or func(t, y, ...)]

        where y can be a vector.

        .. note:: By default, the required order of the first two arguments of
                  `func` are in the opposite order of the arguments in the system
                  definition function used by the `scipy.integrate.ode` class and
                  the function `scipy.integrate.solve_ivp`.  To use a function with
                  the signature ``func(t, y, ...)``, the argument `tfirst` must be
                  set to ``True``.

        Parameters
        ----------
        func : callable(y, t, ...) or callable(t, y, ...)
            Computes the derivative of y at t.
            If the signature is ``callable(t, y, ...)``, then the argument
            `tfirst` must be set ``True``.
        y0 : array
            Initial condition on y (can be a vector).
        t : array
            A sequence of time points for which to solve for y.  The initial
            value point should be the first element of this sequence.
            This sequence must be monotonically increasing or monotonically
            decreasing; repeated values are allowed.
        args : tuple, optional
            Extra arguments to pass to function.
        Dfun : callable(y, t, ...) or callable(t, y, ...)
            Gradient (Jacobian) of `func`.
            If the signature is ``callable(t, y, ...)``, then the argument
            `tfirst` must be set ``True``.
        col_deriv : bool, optional
            True if `Dfun` defines derivatives down columns (faster),
            otherwise `Dfun` should define derivatives across rows.
        full_output : bool, optional
            True if to return a dictionary of optional outputs as the second output
        printmessg : bool, optional
            Whether to print the convergence message
        tfirst: bool, optional
            If True, the first two arguments of `func` (and `Dfun`, if given)
            must ``t, y`` instead of the default ``y, t``.

            .. versionadded:: 1.1.0

        Returns
        -------
        y : array, shape (len(t), len(y0))
            Array containing the value of y for each desired time in t,
            with the initial value `y0` in the first row.
        infodict : dict, only returned if full_output == True
            Dictionary containing additional output information

            =======  ============================================================
            key      meaning
            =======  ============================================================
            'hu'     vector of step sizes successfully used for each time step.
            'tcur'   vector with the value of t reached for each time step.
                     (will always be at least as large as the input times).
            'tolsf'  vector of tolerance scale factors, greater than 1.0,
                     computed when a request for too much accuracy was detected.
            'tsw'    value of t at the time of the last method switch
                     (given for each time step)
            'nst'    cumulative number of time steps
            'nfe'    cumulative number of function evaluations for each time step
            'nje'    cumulative number of jacobian evaluations for each time step
            'nqu'    a vector of method orders for each successful step.
            'imxer'  index of the component of largest magnitude in the
                     weighted local error vector (e / ewt) on an error return, -1
                     otherwise.
            'lenrw'  the length of the double work array required.
            'leniw'  the length of integer work array required.
            'mused'  a vector of method indicators for each successful time step:
                     1: adams (nonstiff), 2: bdf (stiff)
            =======  ============================================================

        Other Parameters
        ----------------
        ml, mu : int, optional
            If either of these are not None or non-negative, then the
            Jacobian is assumed to be banded.  These give the number of
            lower and upper non-zero diagonals in this banded matrix.
            For the banded case, `Dfun` should return a matrix whose
            rows contain the non-zero bands (starting with the lowest diagonal).
            Thus, the return matrix `jac` from `Dfun` should have shape
            ``(ml + mu + 1, len(y0))`` when ``ml >=0`` or ``mu >=0``.
            The data in `jac` must be stored such that ``jac[i - j + mu, j]``
            holds the derivative of the `i`th equation with respect to the `j`th
            state variable.  If `col_deriv` is True, the transpose of this
            `jac` must be returned.
        rtol, atol : float, optional
            The input parameters `rtol` and `atol` determine the error
            control performed by the solver.  The solver will control the
            vector, e, of estimated local errors in y, according to an
            inequality of the form ``max-norm of (e / ewt) <= 1``,
            where ewt is a vector of positive error weights computed as
            ``ewt = rtol * abs(y) + atol``.
            rtol and atol can be either vectors the same length as y or scalars.
            Defaults to 1.49012e-8.
        tcrit : ndarray, optional
            Vector of critical points (e.g. singularities) where integration
            care should be taken.
        h0 : float, (0: solver-determined), optional
            The step size to be attempted on the first step.
        hmax : float, (0: solver-determined), optional
            The maximum absolute step size allowed.
        hmin : float, (0: solver-determined), optional
            The minimum absolute step size allowed.
        ixpr : bool, optional
            Whether to generate extra printing at method switches.
        mxstep : int, (0: solver-determined), optional
            Maximum number of (internally defined) steps allowed for each
            integration point in t.
        mxhnil : int, (0: solver-determined), optional
            Maximum number of messages printed.
        mxordn : int, (0: solver-determined), optional
            Maximum order to be allowed for the non-stiff (Adams) method.
        mxords : int, (0: solver-determined), optional
            Maximum order to be allowed for the stiff (BDF) method.

        See Also
        --------
        solve_ivp : Solve an initial value problem for a system of ODEs.
        ode : a more object-oriented integrator based on VODE.
        quad : for finding the area under a curve.

        Examples
        --------
        The second order differential equation for the angle `theta` of a
        pendulum acted on by gravity with friction can be written::

            theta''(t) + b*theta'(t) + c*sin(theta(t)) = 0

        where `b` and `c` are positive constants, and a prime (') denotes a
        derivative.  To solve this equation with `odeint`, we must first convert
        it to a system of first order equations.  By defining the angular
        velocity ``omega(t) = theta'(t)``, we obtain the system::

            theta'(t) = omega(t)
            omega'(t) = -b*omega(t) - c*sin(theta(t))

        Let `y` be the vector [`theta`, `omega`].  We implement this system
        in python as:

        >>> def pend(y, t, b, c):
        ...     theta, omega = y
        ...     dydt = [omega, -b*omega - c*np.sin(theta)]
        ...     return dydt
        ...

        We assume the constants are `b` = 0.25 and `c` = 5.0:

        >>> b = 0.25
        >>> c = 5.0

        For initial conditions, we assume the pendulum is nearly vertical
        with `theta(0)` = `pi` - 0.1, and is initially at rest, so
        `omega(0)` = 0.  Then the vector of initial conditions is

        >>> y0 = [np.pi - 0.1, 0.0]

        We will generate a solution at 101 evenly spaced samples in the interval
        0 <= `t` <= 10.  So our array of times is:

        >>> t = np.linspace(0, 10, 101)

        Call `odeint` to generate the solution.  To pass the parameters
        `b` and `c` to `pend`, we give them to `odeint` using the `args`
        argument.

        >>> from scipy.integrate import odeint
        >>> sol = odeint(pend, y0, t, args=(b, c))

        The solution is an array with shape (101, 2).  The first column
        is `theta(t)`, and the second is `omega(t)`.  The following code
        plots both components.

        >>> import matplotlib.pyplot as plt
        >>> plt.plot(t, sol[:, 0], 'b', label='theta(t)')
        >>> plt.plot(t, sol[:, 1], 'g', label='omega(t)')
        >>> plt.legend(loc='best')
        >>> plt.xlabel('t')
        >>> plt.grid()
        >>> plt.show()

```python
# Modelo de capitalización continua
def cap_continua(C, t, r):
    return r * C
```

```python
def interes_continuo(C_0, meta, r):
    # Despejamos t
    t = np.log(meta / C_0) / r
    # Vector de periodos
    tt = np.linspace(0, t, 100)
    # Vector de capitales por periodo
    CC = odeint(cap_continua, C_0, tt, args = (r,))
    # Gráfico
    plt.figure(num=1)  # Figura 1
    plt.clf()  # Borrar lo que contenga
    plt.plot(tt, CC, '-', linewidth=3)  # Se grafica la evolución de los capitales
    plt.plot(tt,meta*np.ones(len(tt)),'--k')  # Se grafica la meta
    plt.xlabel('t')  # Etiqueta eje x
    plt.ylabel('C(t)')  # Etiqueta eje y
    plt.grid()  # Malla en la gráfica
    plt.show()  # Mostrar la figura
    print("El tiempo que se debe dejar invertido el dinero para llegar a la meta de ", meta," es ", t, " meses.", sep="")
```

```python
interact_manual(interes_continuo,
                C_0=fixed(10000),
                meta=(10000,12000,100),
                r=fixed(np.log(1+0.008)));
```

    interactive(children=(IntSlider(value=11000, description='meta', max=12000, min=10000, step=100), Button(descr…

### Actividad.
1. Resolver simbólicamente y numéricamente la siguiente ecuación diferencial:
$$\frac{dx}{dt}=\frac{x}{t+1}; \quad x(0) = 1$$

```python
# Definimos variables y función

# Ecuacion diferencial

```

```python

```

2. Obtener la solución numérica.

```python
# Funcion a integrar

```

```python
# Condicion inicial

# Vector de tiempo

```

```python
# Solucion numerica

```

```python
# Grafica x vs.t

```

3. Comparar.

```python
# Grafica x vs.t

```

___
## Tabla de abonos
Como aplicación importante del concepto de interés compuesto se encuentra la creación de un modelo de cuenta de ahorro.

Referencia:
- Vidaurri Aguirre, Héctor Manuel. *Ingeniería económica básica*, ISBN: 978-607-519-017-4. (Disponible en biblioteca)
- http://pbpython.com/amortization-model.html

```python
# Librería de análisis de datos
import pandas as pd
# Librería para manipulación de fechas
from datetime import date
```

```python
# Datos para la cuenta de ahorro
Tasa_interes = 0.08  # 8% anual
Anos = 5
Abonos_ano = 12
Inicial = 50000
Meta = 100000
fecha_inicio = date(2021, 5, 1)

# Tasa interés por periodo
i = Tasa_interes / Abonos_ano  # Tasa de interés mensual
# Total de periodos
T = Anos * Abonos_ano  #

# Cálculo de abonos mensuales iguales para llegar a la meta de ahorro en el tiempo deseado
Abono = i * (Meta - Inicial * (1 + i)**T) / ((1 + i)**T - 1)

# Rango de fechas en la tabla
rng = pd.date_range(fecha_inicio, periods=T, freq='MS')
rng.name = "Fecha del abono"

# Columnas de abonos y extracto de cuenta en las fechas respectivas
df = pd.DataFrame(index=rng, columns=['Abono', 'Balance'], dtype='float')

# Para poner el indice con los periodos y no con las fechas
df.reset_index(inplace=True)
df.index += 1
df.index.name = "Periodo"  # Columna correspondiente al periodo

# Los abonos son iguales
df["Abono"] = Abono

# Vector indizador de los periodos de los abonos
index_vector = np.arange(1,len(rng)+1)
# Extracto de la cuenta mes a mes
df["Balance"] = Inicial * (1 + i)**index_vector + Abono * (((1 + i)**index_vector - 1) / i)

# Redondear con dos cifras decimales
df = df.round(2)
df

# La siguiente solución es también válida. Sin embargo, evitar el uso de 'for'

#df.loc[1, "Balance"] = Inicial*(1+Tasa_interes/Abonos_ano)**1 + df.loc[1, "Abono"]

#for i in range(2, len(df)+1):
#    # Get the previous balance as well as current payments
#    prev_balance = df.loc[i-1, "Balance"]

#    df.loc[i, "Balance"] = prev_balance*(1+Tasa_interes/Abonos_ano)**1 + df.loc[i, "Abono"]
```

      Fecha del abono
      Abono
      Balance

      Periodo

      1
      2021-05-01
      347.15
      50680.49

      2
      2021-06-01
      347.15
      51365.51

      3
      2021-07-01
      347.15
      52055.10

      4
      2021-08-01
      347.15
      52749.29

      5
      2021-09-01
      347.15
      53448.10

      6
      2021-10-01
      347.15
      54151.57

      7
      2021-11-01
      347.15
      54859.74

      8
      2021-12-01
      347.15
      55572.62

      9
      2022-01-01
      347.15
      56290.26

      10
      2022-02-01
      347.15
      57012.68

      11
      2022-03-01
      347.15
      57739.92

      12
      2022-04-01
      347.15
      58472.01

      13
      2022-05-01
      347.15
      59208.97

      14
      2022-06-01
      347.15
      59950.85

      15
      2022-07-01
      347.15
      60697.68

      16
      2022-08-01
      347.15
      61449.48

      17
      2022-09-01
      347.15
      62206.30

      18
      2022-10-01
      347.15
      62968.16

      19
      2022-11-01
      347.15
      63735.10

      20
      2022-12-01
      347.15
      64507.15

      21
      2023-01-01
      347.15
      65284.35

      22
      2023-02-01
      347.15
      66066.74

      23
      2023-03-01
      347.15
      66854.33

      24
      2023-04-01
      347.15
      67647.18

      25
      2023-05-01
      347.15
      68445.32

      26
      2023-06-01
      347.15
      69248.77

      27
      2023-07-01
      347.15
      70057.58

      28
      2023-08-01
      347.15
      70871.79

      29
      2023-09-01
      347.15
      71691.42

      30
      2023-10-01
      347.15
      72516.51

      31
      2023-11-01
      347.15
      73347.11

      32
      2023-12-01
      347.15
      74183.24

      33
      2024-01-01
      347.15
      75024.95

      34
      2024-02-01
      347.15
      75872.27

      35
      2024-03-01
      347.15
      76725.24

      36
      2024-04-01
      347.15
      77583.89

      37
      2024-05-01
      347.15
      78448.27

      38
      2024-06-01
      347.15
      79318.42

      39
      2024-07-01
      347.15
      80194.36

      40
      2024-08-01
      347.15
      81076.14

      41
      2024-09-01
      347.15
      81963.80

      42
      2024-10-01
      347.15
      82857.38

      43
      2024-11-01
      347.15
      83756.91

      44
      2024-12-01
      347.15
      84662.45

      45
      2025-01-01
      347.15
      85574.02

      46
      2025-02-01
      347.15
      86491.66

      47
      2025-03-01
      347.15
      87415.43

      48
      2025-04-01
      347.15
      88345.35

      49
      2025-05-01
      347.15
      89281.47

      50
      2025-06-01
      347.15
      90223.83

      51
      2025-07-01
      347.15
      91172.48

      52
      2025-08-01
      347.15
      92127.45

      53
      2025-09-01
      347.15
      93088.79

      54
      2025-10-01
      347.15
      94056.53

      55
      2025-11-01
      347.15
      95030.73

      56
      2025-12-01
      347.15
      96011.42

      57
      2026-01-01
      347.15
      96998.65

      58
      2026-02-01
      347.15
      97992.46

      59
      2026-03-01
      347.15
      98992.89

      60
      2026-04-01
      347.15
      100000.00

```python
df['Abono'].sum()
```

$\displaystyle 20829.000000000007$

```python
df['Balance'].plot();
```

> **Tarea.**
> - Averiguar tasas de interés reales en algún banco y proyectar un ahorro mensual para que al terminar su carrera tengan $50000 MXN en su cuenta.
> - Hacer esto en un nuevo archivo, llamarlo Tarea6_ApellidoNombre.ipynb y subirlo a moodle.
> - Plazo: Miércoles.

Created with Jupyter by Esteban Jiménez Rodríguez.


# Characterization of Discrete Systems in the Spectral Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Magnitude and Phase

The discrete-time Fourier domain transfer function $H(e^{j \Omega})$ characterizes the transmission properties of a linear time-invariant (LTI) system with respect to an [harmonic exponential signal](../discrete_signals/standard_signals.ipynb#Complex-Exponential-Signal) $e^{j \Omega k}$ with normalized frequency $\Omega$. In order to investigate the characteristics of an LTI system, often the magnitude $| H(e^{j \Omega}) |$ and phase $\varphi_H(e^{j \Omega})$ of the transfer function are regarded separately. Decomposing the output signal $Y(e^{j \Omega}) = X(e^{j \Omega}) \cdot H(e^{j \Omega})$ into its magnitude $| Y(e^{j \Omega}) |$ and phase $\varphi_Y(e^{j \Omega})$ yields

\begin{align}
| Y(e^{j \Omega}) | &= | X(e^{j \Omega}) | \cdot | H(e^{j \Omega}) | \\
\varphi_Y(e^{j \Omega}) &= \varphi_X(e^{j \Omega}) + \varphi_H(e^{j \Omega})
\end{align}

where $X(e^{j \Omega})$ denotes the input signal, and $| X(e^{j \Omega}) |$ and $\varphi_X(e^{j \Omega})$ its magnitude and phase, respectively. It can be concluded, that the magnitude $| H(e^{j \Omega}) |$ quantifies the frequency-dependent attenuation of the magnitude $| X(e^{j \Omega}) |$ of the input signal by the system, while $\varphi_H(e^{j \Omega})$ quantifies the introduced phase-shift.

A rational transfer function $H(z)$ which is composed from polynomials in $z^{-1}$ can be expressed [in terms of its poles and zeros](../z_transform/definition.ipynb#Representation). Applying this representation to the transfer function $H(e^{j \Omega})$ in the discrete-time Fourier domain yields

\begin{equation}
H(e^{j \Omega}) = K \cdot \frac{\prod_{\mu=0}^{Q} (e^{j \Omega} - z_{0 \mu})}{\prod_{\nu=0}^{P} (e^{j \Omega} - z_{\infty \nu})}
\end{equation}

where $z_{0 \mu}$ and $z_{\infty \nu}$ denote the $\mu$-th zero and $\nu$-th pole of $H(z)$, and $Q$ and $P$ the total number of zeros and poles, respectively. Often the logarithmic magnitude of the transfer function $20 \log_{10} | H(e^{j \Omega}) |$ in [decibels](https://en.wikipedia.org/wiki/Decibel) (dB) is considered. The representation of amplitudes in dB is beneficial due to its clear coverage of wide amplitude ranges and convenient calculus for scaled amplitudes. Using the representation of the transfer function with respect to its poles and zeros yields for the logarithm of the magnitude and the phase

\begin{align}
\log_{10} | H(e^{j \Omega}) | &= \sum_{\mu=0}^{Q} \log_{10} |e^{j \Omega} - z_{0 \mu}| - \sum_{\nu=0}^{P} \log_{10} |e^{j \Omega} - z_{\infty \nu}| + \log_{10} |K|   \\
\varphi_H(e^{j \Omega}) &= \sum_{\mu=0}^{Q} \arg (e^{j \Omega} - z_{0 \mu}) - \sum_{\nu=0}^{P} \arg (e^{j \Omega} - z_{\infty \nu})
\end{align}

where $\arg(\cdot)$ denotes the [argument](https://en.wikipedia.org/wiki/Argument_%28complex_analysis%29) (phase) of a complex function. It can be concluded, that the individual contributions of the poles and zeros to the logarithm of the magnitude and phase can be superimposed. This fact may be exploited to estimate the frequency and phase response of discrete-time systems by considering the influence of individual poles and zeros. This is discussed in the following for a system composed from one pole and one zero ($P=Q=0$).

The magnitude of the transfer function depends on the ratio of Euclidean distances from the pole and zero to the position $e^{j \Omega}$ on the unit circle $|z| = 1$. Small distances, e.g. a pole or zero close to the unit circle, lead to large negative values of the individual summands in the logarithmic magnitude response, e.g. $\log_{10} |e^{j \Omega} - z_{0 \mu}|$. This holds especially for normalized frequencies $\Omega$ in the proximity of the pole/zero. It can be concluded that

* a pole close to the unit circle results in a resonance (e.g. maximum), and
* a zero close to the unit circle results in a notch/anti-resonance (e.g. minimum)

in the magnitude response. The normalized frequency $\Omega$ of this maximum/minimum is given by the argument of the pole $\arg \{ z_{\infty \mu} \}$ and zero $\arg\{ z_{0 \mu} \}$, respectively.

**Example - Second order system**

The magnitude response of a real-valued second-order LTI system with the specific transfer function

\begin{equation}
H(e^{j \Omega}) = \frac{(e^{j \Omega} - z_0)(e^{j \Omega} - z_0^*)}{(e^{j \Omega} - z_\infty)(e^{j \Omega} - z_\infty^*)}
\end{equation}

is considered. It is convenient to represent the pole/zero by its magnitude and phase in order to illustrate its location relative to the unit circle

\begin{align}
z_0 &= r_0 e^{j \Omega_0}  &\text{with } \quad &r_0 = 0.8 & \Omega_0 = \frac{\pi}{2} \\
z_\infty  &= r_\infty e^{j \Omega_\infty} & &r_\infty = 0.95 & \Omega_\infty = \frac{\pi}{4}
\end{align}

where for instance $r_0 = |z_0|$ and $\Omega_0 = \arg \{ z_0 \}$. First the contribution of the pair of complex conjugate zeros $z_0$ and $z_0^*$ to the logarithmic magnitude of the transfer function $H(e^{j \Omega})$ is computed and plotted over the normalized frequency $\Omega$.

```python
import sympy as sym
%matplotlib inline
sym.init_printing()

def db(x):
    'compute dB value'
    return 20 * sym.log(sym.Abs(x), 10)

W = sym.symbols('Omega', real=True)
z_0 = 0.8 * sym.exp(sym.I * sym.pi/2)

H1 = (sym.exp(sym.I * W) - z_0)*(sym.exp(sym.I * W) - sym.conjugate(z_0))
Hlog1 = db(H1)
sym.plot(Hlog1, (W, 0, sym.pi), xlabel='$\Omega$',
         ylabel='$|H_0(e^{j \Omega})|$ in dB')
```

    <sympy.plotting.plot.Plot at 0x118e48ba8>

Now the contribution of the pair of complex conjugate poles $z_\infty$ and $z_\infty^*$ is computed and plotted

```python
z_inf = 0.95 * sym.exp(sym.I * sym.pi/4)

H2 = 1/((sym.exp(sym.I * W) - z_inf)*(sym.exp(sym.I * W) - sym.conjugate(z_inf)))
Hlog2 = db(H2)
sym.plot(Hlog2, (W, 0, sym.pi), xlabel='$\Omega$',
         ylabel='$|H_\infty(e^{j \Omega})|$ in dB')
```

    <sympy.plotting.plot.Plot at 0x1193a1ef0>

The logarithmic magnitude response of the system is given by superposition of the individual contributions from the zeros and the poles

```python
Hlog = Hlog1 + Hlog2
sym.plot(Hlog, (W, 0, sym.pi), xlabel='$\Omega$',
         ylabel='$|H(e^{j \Omega})|$ in dB')
```

    <sympy.plotting.plot.Plot at 0x1194a4d68>

**Exercise**

* Why is the system real valued?
* Examine the magnitude response of the system. How is it related to the magnitude responses of the individual zeros/poles?
* Move the poles and/or zeros closer to the unit circle by changing the values $r_\infty$ and/or $r_0$. What influence does this have on the magnitude response?
* Change the phase of the pole and/or zero by changing the values $\Omega_\infty$ and/or $\Omega_0$. What influence does this have on the magnitude response?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# 521153S, Deep Learning assignment 2: Backpropagation, gradient check using linear approximation

## Outline
#### In this assignment, you will learn:
* **Backpropagation**.
* Building a one layer NN using numpy.
* Gradient check using finite-difference approximation.
* Stochatic Gradient Descent (SGD).
* Simple hype-parameters tuning methods to improve your NN performance.

#### Grading (15 points)
In this assignment, we are going to learn about **backpropagation**. This is a very importance algorithm you need to know when building and training your neural network.
* **Part 1.** Import libraries, loading and preprocessing the training and testing data.
* **Part 2.** Backpropagation. (10 points)
  * 2.1. Construct model in **Fig. 2** (2 points)

       * 2.1.1 Declare W1 and W2 (0.5 point)

       * 2.1.2 Implement sigmoid (0.5 point)

       * 2.1.3 Implement softmax (0.5 point)

       * 2.1.4 Implement the cross-entropy loss (0.5 point)

  * 2.2. Implement the forward-pass (1.5 point)

       * 2.2.1 Foward-pass in training (0.5 point)

       * 2.2.2 Call cross entropy loss in training (0.5 point)

       * 2.2.3 Forward-pass in testing (0.5 point)

  * 2.3. Implement SGD (1 point)

       * 2.3.1 Velocity (0.5 point)

       * 2.3.2 Update weights (0.5 point)

  * 2.4. Derive the derivatives (3.0 points)

       * Derive equation 6->12, which Eq. 6+7 (0.5 point), and from Eq. 8-12 (0.5 point) each

  * 2.5. Implement the derivatives (2.5 points)

       * 2.5.1 Implement equation 8 (0.5 point)

       * 2.5.2 Implement equation 6 (0.5 point)

       * 2.5.3 Implement equation 10 (0.5 point)

       * 2.5.4 Implement equation 10+11 (0.5 point)

       * 2.5.5 Implement equation 7 (0.5 point)

* **Part 3.** Gradient check using finite-difference approximation. (3 points)

  * 3.1. Implement gradient check for the model in **Fig. 2** (2.5 points)

       * 3.1.1 Implement reshape_params function (0.5 point)

       * 3.1.2 Adding the epsilon to params then reshape and split flatten_params to hidden weights (W1) and output weights (W2) (0.5 point)

       * 3.1.3 Implement the forward-pass (0.125 point)

       * 3.1.4 Call cross entropy loss in gradient check (0.125 point)

       * 3.1.5 Substracting the epsilon and do the similar calculations as above (0.5 point)

       * 3.1.6 Implement the forward-pass (0.125 point)

       * 3.1.7 Call cross entropy loss in gradient check (0.125 point)

       * 3.1.8 Compute the numerical gradient using the idea in the toy example (0.5 point)

  * 3.2. Question: why don't we use FDA to calculate the gradient to update our model? (0.5 point)

* **Part 4.** Regularization and NN simple tunning. (2 points)

  * 4.1. Applying weight decay. (1 point)

       * There are 5 spots you need to fill, (0.2 point) each spot.
  * 4.2. Change the number of neurons in the hidden layer and report the performance (1 point)

#### Environment
Python 3, Numpy, matplotlib, sklearn

#### Dataset
* [**Fashion-MNIST**](https://github.com/zalandoresearch/fashion-mnist)
is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. We intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits. Using the Fashion-MNIST give you more room to wiggle your experiments.

#### Hints
* To find the place where you have to insert your solution, hit Crtl + F and search for **TODO:** . You are NOT suppose to modify the codes from other parts.
* **Be careful with the shape** of the weights, gradient, .. of your tensor in your implementation. Double check and make sure the shapes are fit for computation, especially matrix multiplication.

## Part 1. Import libraries, loading and preprocessing the training and testing data
**You don't need to change the code from this part.**

```python
# You will mainly use numpy to construct your NN
import numpy as np
import matplotlib, time, copy, os, requests, zipfile, sys
# Matplotlib to plot the image
import matplotlib.pyplot as plt
# Off-the-shelf evaluation functions provided by sklearn
from sklearn.metrics import confusion_matrix, accuracy_score
# Matplotlib predefined 'magic function'. It will include your graphs in your notebook, next to the code
%matplotlib inline
```

### Functions use to download the dataset from google drive
The code snipet was taken from [this thread](https://stackoverflow.com/questions/38511444/python-download-files-from-google-drive-using-url).

```python
def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"

    session = requests.Session()

    response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value

    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768

    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)

```

### Functions use to pre-process your training/testing data

```python
def reshape_train_data(X):
    ''' Input training data has shape (60000, 28, 28)
        Input testing data has shape (10000, 28, 28)
        where:
        60000 is the numbers of input training samples
        10000 is the numbers of input testing samples
        similar to MNIST, resolution of each sample is 28 x 28
    '''
    samples, H, W = X.shape
    # Reshape input volume to (sample, 784), this mean, your NN input layer will have 784 placeholders
    # we scale the RGB values by divide them by 255, this will help improve the training performance
    return X.reshape(samples, H * W).T / 255

def one_hot_vector(x, num_classes):
    # By now, I think you already heard about this so many times
    return np.eye(num_classes)[x].T
```

### We took care of download the data for you.
The fashion-MNIST data will be download and store in your **work_dir/data/fashion_mnist_npy**

```python
PATH = './data'
if not os.path.exists(PATH):
    os.makedirs(PATH)

    file_id = '1DQ2Nf2rY467kyZKOf_CG3Kib5FLv0xQu'
    destination = os.path.join(PATH, 'fashion_mnist_npy.zip')
    download_file_from_google_drive(file_id, destination)

    with zipfile.ZipFile(destination, 'r') as zip_ref:
        zip_ref.extractall(PATH)

    print("Data downloaded and extracted!")

    os.remove(destination)

else:
    print("Data was already downloaded and extracted!")

PATH = os.path.join(PATH, 'fashion_mnist_npy')

# The actual meaning of the label of your classes.
# E.g. if a output one-hot vector is [0. 0. 0. 1. 0. 0. 0. 0. 0. 0.], it used to prepresent a Dress
label_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
```

### Finally, we load and preprocess the data

```python
# Load the training input
X_train = np.load(os.path.join(PATH, 'train_data.npy'))
# Load the training labels
X_test = np.load(os.path.join(PATH, 'test_data.npy'))
# Load the testing input
Y_train = np.load(os.path.join(PATH, 'train_labels.npy'))
# Load the testing labels
Y_test = np.load(os.path.join(PATH, 'test_labels.npy'))

# Get the number of classes
num_classes = len(label_names)
# Get the number of training samples and their resolution for reshape
num_trains, HEIGHT, WIDTH = X_train.shape

# Reshape the training and testing inputs
X_train, X_test = reshape_train_data(X_train), reshape_train_data(X_test)

# Create one-hot vector for the training and testing labels
Y_train, Y_test = one_hot_vector(Y_train, num_classes), one_hot_vector(Y_test, num_classes)

# This part use to randomly load some of the training and testing image and the one-hot vectors for checking
fig_train, ax_train = plt.subplots(figsize=(16, 8), nrows=1, ncols=5)
fig_train.suptitle("Random image from the TRAINING set", y=0.73, fontsize=16, fontweight='bold')

fig_test, ax_test = plt.subplots(figsize=(16, 8), nrows=1, ncols=5)
fig_test.suptitle("Random image from the TESTING set", y=0.73, fontsize=16, fontweight='bold')

for idx in range(5):
    i, j = np.random.randint(num_trains), np.random.randint(X_test.shape[0])

    ax_train[idx].imshow(X_train[:,i].reshape(HEIGHT, WIDTH), cmap = matplotlib.cm.binary)
    ax_train[idx].set_title(label_names[np.argmax(Y_train[:,i])] + "\n" + str(Y_train[:,i]))
    ax_train[idx].axis('off')

    ax_test[idx].imshow(X_test[:,j].reshape(HEIGHT, WIDTH), cmap = matplotlib.cm.binary)
    ax_test[idx].set_title(label_names[np.argmax(Y_test[:,j])] + "\n" + str(Y_test[:,j]))
    ax_test[idx].axis('off')

```

## Part 2. Backpropagation (10 points)
#### The fun part start from here.
*Definition:* backpropagation is an algorithm used to **calculate the gradient of a loss function** (sometime called cost function), by applying the chain rule, so that we can update the weights (and bias) of our neural network using optimizer such as gradient descent.

It could be interpreted as the backward propagation (oppose to the forward propagation) of the network’s layers error, in which we calculate the error of the current layer, then pass the weighted error back to the previous layer, and recursively doing this until we travel back to the first hidden layer. At each layer, we update the weights using the derivative of the cost for each weight.

Let illustrate this concept using a simple toy example.

   Figure 1. Toy example of backpropagation

In **Fig. 1** we have a loss function $L$:
\begin{align}
L = c \times d
\end{align}

where

\begin{align}
c = a + b -5
\end{align}

and,

\begin{align}
d = b^2 + b -1
\end{align}

As illustrated in **Fig. 1**, the equations in the edge show the partial derivation of some functions with respect to their direct variables. For example, $\frac{\partial L}{\partial c}$ is the partial derivative of $L$ with respect to $c$.

However, we are actually interested in calculate the derivative of $L$ with respect to $a$ and $b$, which don't directly connected with each other. So, how can we do this? As mentioned earlier, we can do this using the chain-rule so that we can calculate $\frac{\partial L}{\partial a}$ and $\frac{\partial L}{\partial b}$ as:

\begin{align}
\frac{\partial L}{\partial a} = \frac{\partial L}{\partial c} \cdot \frac{\partial c}{\partial a}
\end{align}

and,

\begin{align}
\frac{\partial L}{\partial b} = \frac{\partial L}{\partial c} \cdot \frac{\partial c}{\partial b} + \frac{\partial L}{\partial d} \cdot \frac{\partial d}{\partial b}
\end{align}

More importantly, **we can see that $a$ affect $L$ through $c$, and so on**. This concept hold no matter how many hidden layers you have or how complicated your loss will be as long as you use backpropagation to calculate the derivative to update your weights.

Now, let's get your hand dirty. Let's take a look at the model in **Fig. 2** below. This is the model we are gonna use in this assignment.

   Figure 2. Structure of our neural network

### Construct model base on **Fig. 2** above (2 points)
As shown in the **Fig. 2**, our neural network contains one hidden layer (we use 5 neurons as default setting) and an output layer. To further simply things, we will not include biases in our model. Be aware that, while the number of neuron in the hidden layer can be abitrary, there must be 10 neurons in the output layer because we want our model to classify images from 10 classes. Based on the **Fig. 2**, you have to construct your network. Don't worry, it's easy and you already learnt how to do this in assignment 1

```python
# Numbers of input units
num_input = X_train.shape[0]
# Number of neural in your hidden layer
# TODO: modify the number of neurons in the hidden layer
num_hidden = 5

# Construct your neural network from Fig. 2 (0.5 point)
# TODO: Random initialize the hidden layer weights (w/o bias)
W1 = np.random.randn(num_hidden, num_input)
# TODO: Random initialize the output layer weights (w/o bias)
W2 = np.random.randn(num_classes, num_hidden)

### Define CrossEntropyLoss in nn
criterion = nn.CrossEntropyLoss()

def sigmoid(X):
    # TODO: implement Eq. 2 (0.5 point)
    return 1 / (1 + np.exp(-X))

def softmax(X):
    # TODO: implement Eq. 4 (0.5 point)
    e_x = np.exp(X)
    return e_x / np.sum(e_x)

def cross_entropy_loss(Y, Y_pred):
    # TODO: implement Eq. 5 (0.5 point)
    m = Y.shape[1]
    log_likelihood = -np.log(Y_pred)*Y
    loss = np.sum(log_likelihood)/m
    return loss
```

According to the model in **Fig. 2**, our **forward-propagation** is going to start with the calculation of the output volume $A_1$ for the hidden layer:

\begin{align}
A_1 = \sigma(Z_1) = \sigma(W_1 X ) \tag{1}
\end{align}
where $W_1$ is the weights of the hidden layer, $X$ is the input, and $\sigma$ is the sigmoid activation where:

\begin{align}
\sigma(x) = \frac{1}{1 + e^{-x}} \tag{2}
\end{align}

Then, we calculate the output volume of the output layer $A_2$ by:

\begin{align}
A_2 = S(Z_2) = S(W_2 A_1) \tag{3}
\end{align}

In **Eq. 3**, $W_2$ is the weights of the output layer, $A_1$ is the output volume of the hidden layer, $S$ stands for softmax and defines by:

\begin{align}
S(x_i) = \frac{e^{x_i}}{\sum_{j=0}^{k} e^{x_j} } \tag{4}
\end{align}
where $i=0,1,..,k$. We use $k$ to represent classes, and $k=9$ in our case.

Finally, we compute the loss function using:

\begin{align}
L(Y, A_2) = - \frac{1}{M} \sum_{k=0}^{M} \sum_{i=0}^{N} Y_i^{k} log({A_2}_i^{k}) \tag{5}
\end{align}

where $Y$ is the ground truth labels, $N$ is the number of classes, $M$ is the number of samples in the training batch.

### Derive the derivative and fill the equation 6->12 below (3.0 points)

Using what you learnt in undergraduate school, let's calculate the **backward-propagation** for our model in **Fig. 2**.

As mentioned earlier, we are interested in $\frac{\partial L}{\partial W_1}$ and $\frac{\partial L}{\partial W_2}$, where: (0.5 point)

\begin{align}
\frac{\partial L}{\partial W_2} =  \tag{6}
\end{align}

and,

\begin{align}
\frac{\partial L}{\partial W_1} =  \tag{7}
\end{align}

From **Eq. 6** we have:

The derivative of $L$ with respect to $Z_2$: (0.5 point)
\begin{align}
\frac{\partial L}{\partial Z_2} = \frac{\partial L}{\partial A_2} \cdot \frac{\partial A_2}{\partial Z_2} =  \tag{8}
\end{align}

The derivative of $Z_2$ with respect to $W_2$: (0.5 point)
\begin{align}
\frac{\partial Z_2}{\partial W_2} =  \tag{9}
\end{align}

From **Eq. 7** we have:

The derivative of $L$ with respect to $A_1$: (0.5 point)
\begin{align}
\frac{\partial L}{\partial A_1} = \frac{\partial L}{\partial Z_2} \cdot \frac{\partial Z_2}{\partial A_1} =  \tag{10}
\end{align}

The derivative of $A_1$ with respect to $Z_1$: (0.5 point)
\begin{align}
\frac{\partial A_1}{\partial Z_1} =  \tag{11}
\end{align}

The derivative of $Z_1$ with respect to $W_1$: (0.5 point)
\begin{align}
\frac{\partial Z_1}{\partial W_1} =  \tag{12}
\end{align}

## Part. 3 Gradient check using finite-difference approximation. (3 points)

When training deep neural network, there are many things that can go wrong. Until this point, you probably notice that we have been going on and on about the gradient. Why? Because the gradient is very important. Hence, you must making sure that the calculation of your gradient is correct. A correct gradient calculation won't promise your model will converge, but if the calculation was wrong your model will perform very weird. This type of error is hard to debug, so we better prevent it beforehand.

To perform the gradient check, we can calculate gradient using the [finite-difference approximation](https://en.wikipedia.org/wiki/Finite_difference) (FDA), and let's call the output of FDA numerical gradients. Then we compare this numerical gradients with the gradient we calculate from taking the derivative. If the differences between them are small enough, we can assume that the gradient was calculated correctly.

You probably learnt about FDA in your undergraduate, but to refresh your mind, let's have a simple example to see how FDA works. Assume that we have a function $f(x)$ which

\begin{align}
f(x) = \frac{1}{3} x^3 - \frac{1}{2} x^2 + 1 \tag{13}
\end{align}

Then, the derivative $\Delta f$ will be:

\begin{align}
\Delta f = x^2 - x \tag{14}
\end{align}

At $x=2.125$, using **Eq. 14** we have $\Delta f = 2.390625$

If we calculate the numerical gradient using FDA we have:
\begin{align}
\Delta_{num\_grad} f = \frac{f(x + \epsilon) - f(x - \epsilon)}{2 * \epsilon}  \tag{15}
\end{align}

where $\epsilon$ is a very small value (E.g. $\epsilon = 1e-{04}$)

At the same point $x=2.125$, using **Eq. 15** we have $\Delta_{num\_grad} f = 2.3906250033389753$

We can see that, the calculated values of $\Delta f$ and $\Delta_{num\_grad} f$ are very close to each other.

With the same idea, we can check the gradient calculation of our network using FDA. A simple way to do this is:

1) We wiggle (by a very small $\epsilon$ values) the value of our weight for all of the parameters in our model. By all parameters, I mean all of the weights of $W_1$ and $W_2$. E.g. if we use 5 neurons in the hidden layer, the number of parameters in our network is :

$num\_params(net) = num\_params(W_1) + num\_params(W_2) = 28*28*5 + 5*10 = 3970$

so we have to repeat the "wiggling" and calculate the numerical gradient 3970 times. At the end, we have a $num\_grad$ vector that have shape (3970,)

2) Calculate the gradient by taking the derivative. Similarly, we will have a $grad$ vector that also have shape (3970,)

3) Compare $num\_grad$ and $grad$ vectors by:

\begin{align}
grad\_diff = \frac{|grad - num\_grad|_2}{|grad + num\_grad|_2}  \tag{16}
\end{align}

If **grad_diff** is smaller than $1e-{08}$ than we assume that our gradient calculation is correct.

### Question: Why don't we use FDA to calculate the gradient to update our model? (0.5 points)

**Answer:**

### Implement gradient check (2.5 points)

```python
def reshape_params(params, num_input, num_hid, num_classes):
    '''Reshape and split flatten_params to hidden weights (W1) and output weights (W2)
        Input:
            + params: the flatten parameters of W1 and W2
            + num_input: the number of input units
            + num_hid: the number of hidden neurons
            + num_classes: the number of output class
        Output:
            + W1: weights of the hidden layer in shape (num_hid, num_input)
            + W2: weights of the output layer in shape (num_class, num_hid)
    '''

    # TODO: implement this function (0.5 point)

    return W1, W2

def calculate_numerical_gradient(_W1, _W2, X, y, is_weight_decay):
    params_flatten = np.concatenate( (copy.deepcopy(_W1).ravel(), copy.deepcopy(_W2).ravel()) )
    numgrad = np.zeros( len(params_flatten) )
    wiggle = np.zeros( len(params_flatten) )
    num_hidden, num_input = _W1.shape
    num_classes = _W2.shape[0]
    e = 1e-4
    batchsize = X.shape[1]

    # Loop through each parameter in our model
    for p in range(len(params_flatten)):
        wiggle[p] = e
        # TODO: adding the epsilon to params then reshape and split flatten_params to hidden weights (W1) and output weights (W2)
        # (0.5 point)
        params_flatten += wiggle
        W1, W2 = reshape_params(params_flatten, num_input, num_hidden, num_classes)

        # TODO: implement the forward-pass (0.125 point)
        y_pre1 = softmax(np.dot(W2,sigmoid(np.dot(W1,X))))

        if is_weight_decay:
            # TODO: call cross entropy loss with weight decay regularization
            # (0.2 point)

            loss1 = cross_entropy_loss(y, y_pre1) + lmda/2*sum(params_flatten*params_flatten)

        else:
            # TODO: call cross entropy loss (0.125 point)
            loss1 = cross_entropy_loss(y, y_pre1)

        # reset the params values
        params_flatten -= wiggle

        # TODO: substract the epsilon and do the similar calculations as above
        # (0.5 point)
        params_flatten -= wiggle
        W1, W2 = reshape_params(params_flatten, num_input, num_hidden, num_classes)

        # TODO: implement the forward-pass (0.125 point)
        y_pre2 = softmax(np.dot(W2,sigmoid(np.dot(W1,X))))

        if is_weight_decay:
            # TODO: call cross entropy loss with weight decay regularization
            # (0.2 point)
            loss2 = cross_entropy_loss(y, y_pre2) + lmda/2*sum(params_flatten*params_flatten)
        else:
            # TODO: call cross entropy loss (0.125 point)
            loss2 = cross_entropy_loss(y, y_pre2)

        # TODO: Compute the numerical gradient using the idea in the toy example (0.5 point)
        numgrad[p] = (y_pre2 - y_pre1)/(2*e)
        # reset the params values
        params_flatten += wiggle

        # Return the value we changed to zero:
        wiggle[p] = 0

    return numgrad
```

### Hyper-parameters in your training

```python
# Flag use to enable/disable gradient check and weight decay regularization
is_gradient_check = False
is_weight_decay = True

if is_weight_decay:
    # Setting lambda coefficient for weight decay
    lmda = np.exp(-7)

# Seting learning rate and momentum for SGD
learning_rate = 0.25
beta = 0.5

# Seting the number of training epochs
epoch = 50
# Choose your batch size
batch_size = 100
# Calculate the number of training iterations base on the number of training samples and your batch size
num_batchs = num_trains // batch_size

print("Num_trains: {}, num_batchs: {}".format(num_trains, num_batchs))

```

### Training your network (4.5 points)

In this assignment, we will train our model using mini-batch stochatic gradient descent with momentum. To know more about this optimization algorithm, please check out this great [video](https://www.youtube.com/watch?v=k8fTYJPd3_I) from Dr. Andrew Ng.

For this assignment, we will use the implementation from **Eq. 17** and **Eq. 18**

\begin{align}
v_{dW} = \beta v_{dW} + (1 - \beta) dW \tag{17}
\end{align}
\begin{align}
W = W - \alpha v_{dW} \tag{18}
\end{align}

If you are curious, you can modify the hyper-params in the above section at your will.

```python
# Define an interactive ipython figure to display the training loss
%matplotlib notebook
fig = plt.figure("Training loss")
ax = fig.add_subplot(1, 1, 1)
plt.ion()
fig.show()
fig.canvas.draw()

# Log the training loss
loss_log = []

# Zeros initialize the momentum for SGD
V_dW1 = np.zeros(W1.shape)
V_dW2 = np.zeros(W2.shape)

# Training
for i in range(epoch):
    start_t = time.time()

    # Random shuffle training data every training epoch
    np.random.seed(np.random.randint(num_trains))
    indices = np.random.permutation(num_trains)
    X_train_shuffled, Y_train_shuffled = X_train[:, indices], Y_train[:, indices]

    for j in range(num_batchs):

        # Get mini-batch samples for training
        start_idx = j * batch_size
        end_idx = min(j * batch_size + batch_size, X_train.shape[1] - 1)
        X, Y = X_train_shuffled[:, start_idx : end_idx], Y_train_shuffled[:, start_idx : end_idx]
        # Size of actual mini-batch, it could be smaller than batch_size
        mini_batch = end_idx - start_idx

        # TODO: implement the forward-pass (0.5 point)

        if is_weight_decay:
            # TODO: call cross entropy loss with weight decay regularization
            # (0.2 point)

        else:
            # TODO: call cross entropy loss (0.5 point)

        # Log the training loss during training
        loss_log.append(L)

        # Gradient check
        if is_gradient_check:
            numgrad = calculate_numerical_gradient(W1, W2, X, Y, is_weight_decay)

        # TODO: calculate the derivative of 𝐿 with respect to 𝑍2 using eq. 8 (0.5 point)

        if is_weight_decay:
            # TODO: calculate the derivative of 𝐿 with respect to W2 using eq. 6 with weight decay regularization
            # (0.2 point)

        else:
            # TODO: calculate the derivative of 𝐿 with respect to W2 using eq. 6 (0.5 point)

        # TODO: calculate the derivative of 𝐿 with respect to A1 using eq. 10 (0.5 point)

        # TODO: calculate the derivative of 𝐿 with respect to Z1 using eq. 10 and eq. 11 (0.5 point)

        if is_weight_decay:
            # TODO: calculate the derivative of 𝐿 with respect to W1 using eq. 7 with weight decay regularization
            # (0.2 point)

        else:
            # TODO: calculate the derivative of 𝐿 with respect to W1 using eq. 7 (0.5 point)

        # Gradient check
        if is_gradient_check:
            # Flatten and combine grad vector
            grad = np.concatenate((dW1.ravel(), dW2.ravel()))
            # Compare num_grad and grad
            grad_diff = np.linalg.norm(grad - numgrad) / np.linalg.norm(grad + numgrad)
            # print("Grad_diff: {}".format(grad_diff))
            assert grad_diff < 1e-8, "Gradient check fail, please carefully inspect your gradient calculations again!"

        # TODO: Update the learning velocity using Eq. 17 (0.5 point)

        V_dW1 = beta * V_dW1 + (1-beta) * dW1
        V_dW2 = beta * V_dW2 + (1-beta) * dW2

        # TODO: Update the model weights using Eq. 18 (0.5 point)
        W1 -= learning_rate * V_dW1
        W2 -= learning_rate * V_dW2

        if (j % 100 == 0):
            print("[Epoch/Iterations]:[{}/{}], loss: {}".format(i, j, L))

    ax.clear()
    ax.plot(loss_log)
    fig.canvas.draw()
    print("=> Elapsed time epoch #{} : {:.2f} seconds".format(i, time.time() - start_t))

```

### Evaluate the performance of your model (0.5 points)

```python
# TODO: implement the forward-pass (0.5 point)
A2 = softmax(np.dot(W2,sigmoid(np.dot(W1,X_test))))

# Evaluate the performance of your NN
predictions = np.argmax(A2, axis=0)
labels = np.argmax(Y_test, axis=0)
print("Confusion matrix:\n{}".format(confusion_matrix(labels, predictions)))
print("Testing accuracy: {}".format(accuracy_score(labels, predictions)))
```

## Part 4. Regularization and NN simple tunning (2 points)

1. Applying weight decay (1 point)
  * Using what you learnt from assignment 1 to add the code at neccesary parts in **Training your network** and **Implement gradient check** above.
  * There are 5 spots, (0.2 point) each spot.
2. Change the number of neurons in the hidden layer and report the performance (1 point)

**Answer of question 2:**


# Gradient Descent Doesn't Find a Local Minimum in a Two Layer Linear Net

$$\newcommand{\E}{\mathop{\mathbb{E}}}
\newcommand{\L}{\mathop{\mathcal{L}}}$$

I'll fit a two layer neural net on a realizable problem.

The loss function to find matrices $W_1$ and $W_2$ that minimize

$$\begin{align}
\L(W_1, W_2) &= \E_{x \sim \mathcal{N}(0,\; I)} \|W_1 W_2 x - y\|^2 \\
y &= Ax
\end{align}$$

where $A$ is a badly conditioned matrix ($\kappa = 10^5$).

This problem is matrix factorization in thin disguise:
$$\begin{align}
\L &= \E \|(W_1W_2 -A)x\|^2 \\
&= \text{tr}\;  (W_1W_2-A)\left(\E xx^T\right)(W_1W_2-A)^T \\
&= \|W_1W_2-A\|_F^2.
\end{align}$$

# Gradient Descent and Variants

```
import tensorflow as tf
import matplotlib.pylab as plt
import numpy as np
from scipy import sparse
```

```
class TwoLayerLinearNet(object):

  def __init__(self, name, X, Ytrue, W1_initial, W2_initial):
    W1 = tf.Variable(W1_initial, trainable=True)
    W2 = tf.Variable(W2_initial, trainable=True)

    # Yhat = W1 W2 X
    Yhat = tf.matmul(W1, tf.matmul(W2, X))
    self.variables = (W1, W2)

    # loss = || Yhat - Ytrue ||^2
    loss = tf.reduce_sum(tf.squared_difference(Yhat, Ytrue))
    self.loss = loss

    # a tensor that stores the sum of the frobenius norms of the gradients of
    # the loss wrt to the parameters
    self.gradient_magnitude = tf.sqrt(sum([
        tf.reduce_sum(dLdvar**2)
        for dLdvar in tf.gradients(loss, self.variables)
    ]))

    # this optimizer must be set before calling run_optimizer
    self.optimizer_op = None
    self.name = name
    self.losses = []
    self.plot_kwargs = {}

  def run_optimizer(self, sess, num_steps):
    self.losses = []
    self.gradient_magnitudes = []

    for it in xrange(num_steps):
      loss, gradient_magnitude, _ = sess.run([
          self.loss,
          self.gradient_magnitude,
          self.optimizer_op
      ])

      self.losses.append(loss)
      self.gradient_magnitudes.append(gradient_magnitude)

      # stop iterating if the loss has diverged
      if not np.isfinite(self.losses[-1]):
        break

class NetGradientDescent(TwoLayerLinearNet):

  def __init__(self, learning_rate, *args, **kwargs):
    TwoLayerLinearNet.__init__(self, 'GD step=%g' % learning_rate,
                               *args, **kwargs)

    self.optimizer_op = tf.train.GradientDescentOptimizer(
        learning_rate).minimize(self.loss, var_list=self.variables)

class NetADAM(TwoLayerLinearNet):

  def __init__(self, learning_rate, *args, **kwargs):
    TwoLayerLinearNet.__init__(self, 'ADAM step=%g' % learning_rate,
                               *args, **kwargs)

    self.optimizer_op = tf.train.AdamOptimizer(
        learning_rate).minimize(self.loss, var_list=self.variables)
    self.plot_kwargs = {'ls': '--'}
```

```
def train_nets(xdim=6, wdim=6, ydim=10, nsamples=1000,
               A_condition_number=1e-5):
    W1_initial = np.random.randn(ydim, wdim)
    W2_initial = np.random.randn(wdim, xdim)

    # the true map from x to y
    Atrue = np.linspace(1, A_condition_number, ydim
                       ).reshape(-1, 1) * np.random.rand(ydim, xdim)
    # the inputs
    X = np.random.randn(xdim, nsamples)
    # the y's to fit
    Ytrue = Atrue.dot(X)

    nets = [
        NetGradientDescent(learning_rate, X, Ytrue, W1_initial, W2_initial)
        for learning_rate in [1e-6, 1e-5, 1e-4, 1e-3]
    ] + [
        NetADAM(learning_rate, X, Ytrue, W1_initial, W2_initial)
        for learning_rate in [1e-5, 1e-4, 1e-3, 1e-2]
    ]

    with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())

      for net in nets:
        net.run_optimizer(sess, 3000)

    return nets, (X, Ytrue, W1_initial, W2_initial)
nets, initial_params = train_nets()

```

```
def plot_nets(nets, show_to_max=-1):
  fig = plt.figure(figsize=(15, 10))
  ax1 = fig.add_subplot(2,1,1)
  ax2 = fig.add_subplot(2,1,2)

  for net in nets:
    show_to = -1 if show_to_max > len(net.losses) else show_to_max

    if np.isfinite(net.losses[show_to]):
      ax1.semilogy(net.losses[:show_to], label=net.name, **net.plot_kwargs)
      ax2.semilogy(net.gradient_magnitudes[:show_to], label=net.name,
                   **net.plot_kwargs)
    else:
      ax1.semilogy(net.losses[:1], marker='x', label='(diverged) ' + net.name,
                   **net.plot_kwargs)

  ax1.set_ylabel('Loss')
  ax1.legend(loc='best')
  ax2.set_ylabel('Gradient magnitude')
  ax2.legend(loc='best')
  return ax1, ax2

plot_nets(nets, -1);
```

# Is This Really a Problem?

The optimization problem ought to reach $\L = 0$ (the problem is realizable). The loss seems to go down by a few orders of magnitude, but it does not reach 0 to machine precision.

It's tempting to say that the optimization has reached some kind of noise floor for the data set: perhaps adding more data points solves the issue. Not so. Minimizing the expected value of the loss, $\|W_1 W_2 - A\|$, produces similar curves. What we're orbserving has nothing to do with statistics of $\L$. It's entirely about the dynamics of gradient descent on product chains.

Gradient descent hits an optimization floor, not a statistical floor. You don't somehow learn more by adding more data.

# What A Better Optimizer Could Do

Levenberg-Marquardt is the simplest optimizer I can think of that does not require second order information. We'll take steps $(\delta W_1, \delta W_2)$ that minimize an approximation to $\L$:

$$\begin{align}
\L(W_1+\delta W_1, W_2+\delta W_2) &= \E_x \|(W_1+\delta W_1)(W_2+\delta W_2)x - y\|^2 \\
&\approx  \E_x \|(\delta W_1\; W_2+W_1\;\delta W_2)x  + \underbrace{W_1 W_2 x- y}_e\|^2 \\
&=  \E_x \left\|\begin{bmatrix}(W_2x)^T \otimes I, & x^T\otimes W_1\end{bmatrix}\begin{bmatrix}\delta \vec{W}_1 \\ \delta \vec{W}_2\end{bmatrix}  + e\right\|^2.
\end{align}$$

We'll look for the minimizing  $(\delta W_1, \delta W_2)$ by solving this least-squares problem. The algorithm looks a lot like K-FAC or Natural Gradient.

```
def unpack_vector(x, *lengths):
  """Split up a sequence into several sequences of the given length.

  Args:
    x: a sequence
    lengths: a sequence of n lengths.
  Returns:
    A list of sequences v1...vN. vi has length lengths[i].
  """
  vectors = []
  i = 0
  for l in lengths:
    v = x[i:i+l]
    vectors.append(v)
    i += l
  assert i == len(x)
  return vectors

class LevenbergMarquardt(object):

  def __init__(self, learning_rate, X, Ytrue, W1_initial, W2_initial):
    self.learning_rate = learning_rate
    self.name = "LM step=%g" % learning_rate
    self.X = X
    self.Ytrue = Ytrue
    self.W1 = W1_initial
    self.W2 = W2_initial
    self.plot_kwargs = {}

  def take_step(self):
    "Form and solve the least squares problem above about W1, W2."

    Yhat = self.W1.dot(self.W2.dot(self.X))
    RHS = -(Yhat - self.Ytrue).T.flatten()

    LHS = np.hstack((
        np.kron(self.W2.dot(self.X).T, np.eye(self.W1.shape[0])),
        np.kron(self.X.T, self.W1)
    ))

    # solve the least squares problem and parse the solution into matrices
    dW1dW2 = np.linalg.lstsq(LHS, RHS)[0]
    dW1, dW2 = unpack_vector(dW1dW2, self.W1.size, self.W2.size)
    dW1 = dW1.reshape(self.W1.T.shape).T
    dW2 = dW2.reshape(self.W2.T.shape).T

    # apply the update to the iterates
    self.W1 = self.W1 + self.learning_rate * dW1
    self.W2 = self.W2 + self.learning_rate * dW2

  def loss(self):
    Yhat = self.W1.dot(self.W2.dot(self.X))
    return np.sum(Yhat - self.Ytrue)**2

  def run_optimizer(self, num_steps):
    self.gradient_magnitudes = []
    self.losses = []
    for _ in xrange(num_steps):
      self.take_step()
      self.losses.append(self.loss())

def train_lm(X, Ytrue, W1_initial, W2_initial):
  nets = []
  for step_sizes in [.05, .2]:
    net = LevenbergMarquardt(step_sizes, X, Ytrue, W1_initial, W2_initial)
    net.run_optimizer(200)
    nets.append(net)
  return nets

nets_lm = train_lm(*initial_params)
```

```
nets_lm[0].plot_kwargs = {'lw': 5, 'color': 'k', 'alpha': .8}
ax1, _ = plot_nets(nets + nets_lm, 191);
ax1.set_yticks(10**np.linspace(-14, 6, 7))
ax1.axis('tight')
```


# Random Signals and LTI-Systems

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Linear Mean

In the following we aim at finding a relation between the linear mean $\mu_x[k]$ of the input signal $x[k]$ and the linear mean $\mu_y[k]$ of the output signal $y[k] = \mathcal{H} \{ x[k] \}$ of a linear time-invariant (LTI) system.

### Non-Stationary Input Signal

Let's first impose no restrictions in terms of stationarity to the input signal. The [linear mean](../random_signals/ensemble_averages.ipynb#Linear-mean) of the output signal is then given as

\begin{equation}
\mu_y[k] = E\{ y[k] \} = E\{ x[k] * h[k] \}
\end{equation}

where $h[k]$ denotes the impulse response of the system. Since the convolution and the ensemble average are linear operations, and $h[k]$ is a deterministic signal, this can be rewritten as

\begin{equation}
\mu_y[k] = \mu_x[k] * h[k]
\end{equation}

The linear mean of the output signal $\mu_y[k]$ is given as the convolution of the linear mean of the input signal $\mu_x[k]$ with the impulse response $h[k]$ of the system.

#### Example

The linear mean $\mu_y[k]$ of the output of an LTI system with given impulse response $h[k]$ and non-stationary random input signal $x[k]$ is computed. The estimated linear means $\hat{\mu}_x[k]$ and $\hat{\mu}_y[k]$ of the input and output signals are plotted.

```python
import numpy as np
import matplotlib.pyplot as plt

L = 32  # number of random samples
N = 10000  # number of sample functions

# generate input signal (white Gaussian noise)
np.random.seed(2)
x = np.random.normal(size=(N, L))
x[:, L//2] += 1
# generate output signal
h = 2*np.fft.irfft([1, 1, 1, 0, 0, 0])
y = np.asarray([np.convolve(x[n, :], h, mode='full') for n in range(N)])

def estimate_plot_linear_mean(x):
    '''Estimate and plot linear mean.'''
    # estimate linear mean by ensemble average
    mu = 1/N * np.sum(x, 0)
    # plot linear mean
    plt.stem(mu, use_line_collection=True)
    plt.xlabel(r'$k$')
    plt.ylabel(r'$\hat{\mu}[k]$')
    plt.axis([0, x.shape[1], -1.2, 1.2])

plt.figure(figsize=(10, 3))
plt.title(r'Estimated linear mean $\hat{\mu}_x[k]$ of input signal')
estimate_plot_linear_mean(x)

plt.figure(figsize=(10, 3))
plt.title(r'Estimated linear mean $\hat{\mu}_y[k]$ of output signal')
estimate_plot_linear_mean(y)
```

**Exercise**

* Can you estimate the impulse response $h[k]$ of the system from above plots of $\hat{\mu}_x[k]$ and $\hat{\mu}_y[k]$?
* You can check your results by plotting the impulse response $h[k]$, for instance with the command `plt.stem(h)`.

Solution: Inspecting above plot, the linear mean of the input signal can be approximated as $\mu_x[k] = \delta[k]$. The linear mean of the output is then given as $\mu_y[k] = \delta[k] * h[k] = h[k]$. It follows that the impulse response of the LTI system can be estimated from the linear mean $\mu_y[k]$.

### Stationary Input Signal

For a (wide-sense) stationary process, the linear mean of the input signal $\mu_x[k] = \mu_x$ does not depend on the time index $k$. For a (wide-sense) stationary input signal, also the output signal of the system is (wide-sense) stationary. Using the result for the non-stationary case above yields

\begin{equation}
\begin{split}
\mu_y &= \mu_x * h[k] \\
&= \sum_{\kappa = -\infty}^{\infty}\mu_x[k-\kappa]h[\kappa] \\
&= \mu_x \cdot \sum_{\kappa = -\infty}^{\infty}h[\kappa] \\
&= \mu_x \cdot \sum_{\kappa = -\infty}^{\infty}h[\kappa]\cdot\mathrm{e}^{-\mathrm{j}\Omega\kappa} \hspace{5mm} \text{for}\,\,\Omega=0 \\
&= \mu_x \cdot H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \big\vert_{\Omega = 0}
\end{split}
\end{equation}

where $H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathcal{F}_* \{ h[k] \}$ denotes the discrete time Fourier transformation (DTFT) of the impulse response. The linear mean of a (wide-sense) stationary input signal is weighted by the transmission characteristics for the constant (i.e. DC, $\Omega = 0$) component of the LTI system. This implies that for a system which just attenuates the input signal $y[k] = A \cdot x[k]$, e.g. an ideal amplifier, the linear mean at the output is given as $\mu_y = A \cdot \mu_x$. Furthermore, if the input signal is zero-mean $\mu_x = 0$, the output signal is also zero-mean $\mu_y = 0$.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples.