# Ejemplos de Bisección e Iteración de Punto Fijo

```python
import numpy as np
import matplotlib.pyplot as plt
```

Cálculo del error

\begin{equation}
    error = |x - r|
\end{equation}

```python
error = lambda x, r: np.abs(x - r)
```

## Problema

Obtener $\sqrt{2}$

\begin{equation}
    \begin{split}
        x & = \sqrt{2} \quad /()^2\\
        x^2 & = 2
    \end{split}
\end{equation}

Si bien llegamos a una ecuación cuadrática la cual tiene dos soluciones, nos interesa el valor positivo por lo que tanto para la *bisección* como para la *IPF* debemos elegir convenientemente el intervalo y estimación inicial respectivamente. Para este problema debería tomarse en cuenta $x>0$.

## Resolución utilizando Bisección

Se define $f(x)$
\begin{equation}
    f(x)=x^2-2=0
\end{equation}

```python
def bisect(f, a, b, tol=1e-8):
    fa = f(a)
    fb = f(b)
    i = 0
    x = []

    # Just checking if the sign is not negative => not root  necessarily
    if np.sign(f(a)*f(b)) >= 0:
        print('f(a)f(b)<0 not satisfied!')
        return None

    while (b-a)/2 > tol:
        c = (a+b)/2.
        x.append(c)
        fc = f(c)
        # Did we find the root?
        if fc == 0:
            break
        elif np.sign(fa*fc) < 0:
            b = c
            fb = fc
        else:
            a = c
            fa = fc
        i += 1

    #xc = (a+b)/2.
    return np.array(x)#xc
```

```python
f = lambda x: x ** 2 - 2
```

```python
x = np.linspace(0, 2)
plt.figure(figsize=(12, 6))
plt.plot(x, f(x))
plt.grid(True)
plt.xlabel(r"$x$")
plt.ylabel(r"$f(x)$")
plt.show()
```

Buscamos el intervalo donde $f(a)\, f(b) < 0$. En este caso $[a, b]=[1,2]$ parece ser útil.

```python
x_b = bisect(f, 1, 2)
```

```python
n_i = np.arange(1, len(x_b) + 1)
er = error(x_b, np.sqrt(2))
plt.figure(figsize=(12, 6))
plt.plot(n_i, er, 'b.')
plt.grid(True)
plt.xlabel("# iteraciones")
plt.ylabel("Error")
plt.yscale('log')
plt.show()
```

El error decae linealmente a medida que se aumenta el número de iteraciones.

## Resolución utilizando punto Fijo

```python
def cobweb(x, g=None):
    min_x = np.amin(x)
    max_x = np.amax(x)

    plt.figure(figsize=(10,10))
    ax = plt.axes()
    plt.plot(np.array([min_x,max_x]),np.array([min_x,max_x]),'b-')
    for i in np.arange(x.size-1):
        delta_x = x[i+1]-x[i]
        head_length =  np.abs(delta_x)*0.04
        arrow_length = delta_x-np.sign(delta_x)*head_length
        ax.arrow(x[i], x[i], 0, arrow_length, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')
        ax.arrow(x[i], x[i+1], arrow_length, 0, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')

    if g!=None:
        y = np.linspace(min_x,max_x,1000)
        plt.plot(y,g(y),'r')

    plt.title('Cobweb diagram')
    plt.grid(True)
    plt.show()
```

```python
def fpi(g, x0, k, tol=1e-16, flag_cobweb=False):
    x = np.empty(k+1)
    x[0] = x0
    for i in range(k):
        if np.abs(x[i] - x[i-1]) <= tol:
            x = x[:i]
            break
        x[i+1] = g(x[i])

    if flag_cobweb:
        cobweb(x, g)

    return x[1:]
```

### Manejo algebraico:

Debemos construir $g(x)$ para utilizar el algoritmo.

\begin{equation}
    x=g(x)
\end{equation}

#### Version 1

\begin{equation}
    \begin{split}
        x^2 & = 2 \quad / \frac{1}{x}\\
        x & = \frac2x \\
        g_1(x) & = \frac2x
    \end{split}
\end{equation}

#### Version 2

\begin{equation}
    \begin{split}
        x^2 & = 2 \quad / +x\\
        x^2 + x & = 2+x \\
        x & = 2 + x - x^2 \\
        g_2(x) & = 2 + x - x^2
    \end{split}
\end{equation}

#### Version 3

\begin{equation}
    \begin{split}
        x^2 & = 2 \quad / +x^3\\
        x^3 + x^2 & = 2 + x^3 \\
        x(x^2 + x) & = 2 + x^3 \\
        x & = \frac{2+x^3}{x^2 + x} \\
        g_3(x) & = \frac{2+x^3}{x^2 + x}
    \end{split}
\end{equation}

#### Version 4

\begin{equation}
    \begin{split}
        x^2 & = 2 \quad / +x^2\\
        2x^2 & = 2+x^2 \quad  /\frac{1}{2x} \\
        x & = \frac{2 + x^2}{2x} \\
        x & = \frac1x + \frac{x}{2} \\
        g_4(x) & = \frac1x + \frac{x}{2}
    \end{split}
\end{equation}

Algunas funciones se pueden indeterminar para ciertos valores como en el caso de $g_3(x)$ y $g_4(x)$, por lo que hay que tener cuidado con la selección de $x_0$.

```python
g1 = lambda x: 2 / x
g2 = lambda x: 2 + x - x ** 2
g3 = lambda x: (2 + x ** 3) / (x ** 2 + x)
g4 = lambda x: 1 / x + x / 2
```

### Visualización de las funciones $g(x)$

```python
x2 = np.linspace(0.1, 2, 100)
plt.figure(figsize=(12, 6))
plt.plot(x2, g1(x2), label=r"$g_1(x)$")
plt.plot(x2, g2(x2), label=r"$g_2(x)$")
plt.plot(x2, g3(x2), label=r"$g_3(x)$")
plt.plot(x2, g4(x2), label=r"$g_4(x)$")
plt.plot(x2, x2, label=r"$y=x$")
plt.ylim([0, 6])
plt.grid(True)
plt.legend()
plt.show()
```

### Aplicación de IPF

```python
x_f_1 = fpi(g1, 1, 20, flag_cobweb=True)
x_f_1
```

```python
x_f_2 = fpi(g2, 1, 20, flag_cobweb=True)
x_f_2
```

```python
x_f_3 = fpi(g3, 1, 20, flag_cobweb=True)
x_f_3
```

```python
x_f_4 = fpi(g4, 1, 20, flag_cobweb=True)
x_f_4
```

Según la $g(x)$ y $x_0$ que se escoja, la *IPF* puede no converger o iterar alternando entre dos valores.

```python
er_f_1 = error(x_f_1, np.sqrt(2))
er_f_2 = error(x_f_2, np.sqrt(2))
er_f_3 = error(x_f_3, np.sqrt(2))
er_f_4 = error(x_f_4, np.sqrt(2))
plt.figure(figsize=(12, 6))
plt.plot(er_f_1, 'rx', label=r'$g_1(x)$')
plt.plot(er_f_2, 'bs', label=r'$g_2(x)$')
plt.plot(er_f_3, 'yo', label=r'$g_3(x)$')
plt.plot(er_f_4, 'g.', label=r'$g_4(x)$')
plt.grid(True)
plt.xlabel("# iteraciones")
plt.ylabel("Error")
plt.yscale('log')
plt.legend()
plt.show()
```

Notar del gráfico que según el $g(x)$ utilizado podemos obtener una solución mucho más rápido como en el caso de $g_4(x)$.

## Convergencia

### Cota teórica

\begin{equation}
    \frac{\epsilon_{i+1}}{\epsilon_i} = \frac{|x_{i+1}-r|}{|x_{i}-r|}
\end{equation}

Veamos la convergencia de los resultados obtenidos

```python
conv_t = lambda x, r: error(x[1:], r) / error(x[:-1], r)
```

#### Para bisección...

```python
conv_b = conv_t(x_b, np.sqrt(2))
```

```python
print("e_{i+1} \t e_{i} \t\t e_{i+1}/e_{i}")
for i, (e1 ,e2) in enumerate(zip(error(x_b[1:], np.sqrt(2)), error(x_b[:-1], np.sqrt(2)))):
    print(("%.6f \t %.6f \t %.6f")%(e1, e2, conv_b[i]))
```

    e_{i+1} 	 e_{i} 		 e_{i+1}/e_{i}
    0.164214 	 0.085786 	 1.914214
    0.039214 	 0.164214 	 0.238796
    0.023286 	 0.039214 	 0.593836
    0.007964 	 0.023286 	 0.341983
    0.007661 	 0.007964 	 0.962062
    0.000151 	 0.007661 	 0.019717
    0.003755 	 0.000151 	 24.858524
    0.001802 	 0.003755 	 0.479886
    0.000826 	 0.001802 	 0.458086
    0.000337 	 0.000826 	 0.408503
    0.000093 	 0.000337 	 0.276017
    0.000029 	 0.000093 	 0.311481
    0.000032 	 0.000029 	 1.105237
    0.000002 	 0.000032 	 0.047608
    0.000014 	 0.000002 	 9.002369
    0.000006 	 0.000014 	 0.444459
    0.000002 	 0.000006 	 0.375037
    0.000000 	 0.000002 	 0.166798
    0.000001 	 0.000000 	 1.497634
    0.000000 	 0.000001 	 0.166140
    0.000000 	 0.000000 	 1.509509
    0.000000 	 0.000000 	 0.168767
    0.000000 	 0.000000 	 1.462673
    0.000000 	 0.000000 	 0.158160
    0.000000 	 0.000000 	 1.661355

```python
plt.figure(figsize=(12, 6))
plt.plot(conv_b, 'bo')
plt.grid(True)
plt.xlabel("# de iteraciones")
plt.ylabel(r"$\epsilon_{i+1}/\epsilon_i$")
plt.show()
```

El error en la bisección puede diferir bastante entre cada iteración porque para $i$ podría estimar $x_c$ muy cerca de $r$ pero en $i+1$ se podría volver a alejar, dado que solo divide el intervalo a la mitad para generar la siguiente estimación.

#### Para IPF

```python
conv_f_1 = conv_t(x_f_1, np.sqrt(2))
conv_f_2 = conv_t(x_f_2, np.sqrt(2))
conv_f_3 = conv_t(x_f_3, np.sqrt(2))
conv_f_4 = conv_t(x_f_4, np.sqrt(2))
```

```python
plt.figure(figsize=(12, 6))
plt.plot(conv_f_1, label=r'$g_1(x)$')
plt.plot(conv_f_2, label=r'$g_2(x)$')
plt.plot(conv_f_3, label=r'$g_3(x)$')
plt.plot(conv_f_4, label=r'$g_4(x)$')
plt.grid(True)
plt.legend()
plt.xlabel("# de iteraciones")
plt.ylabel(r"$\epsilon_{i+1}/\epsilon_i$")
plt.show()
```

$g_4(x)$ parece tener el mejor comportamiento aunque $g_3(x)$ tampoco sería una opción incorrecta.

### Aproximación en cada iteracion

Podemos utilizar esta aproximación si no conocemos $r$.

\begin{equation}
    \frac{\epsilon_{i+1}}{\epsilon_i} \approx \frac{|x_{i+1}-x_{i}|}{|x_{i}-x_{i-1}|}
\end{equation}

```python
conv = lambda x: error(x[2:], x[1:-1]) / error(x[1:-1], x[:-2])
```

#### Para bisección

```python
conv_b = conv(x_b)
conv_b
```

    array([0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5,
           0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5])

#### Para IPF

```python
conv_f_1 = conv(x_f_1)
conv_f_2 = conv(x_f_2)
conv_f_3 = conv(x_f_3)
conv_f_4 = conv(x_f_4)
```

```python
conv_f_1
```

    array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1.])

```python
conv_f_2
```

    array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
           1.])

```python
conv_f_3
```

    array([0.234151  , 0.18575959, 0.1741862 , 0.17202736, 0.17165104,
           0.17158629, 0.17157518, 0.17157327, 0.17157294, 0.17157288,
           0.17157286, 0.17157226, 0.1715725 , 0.1715798 , 0.17169266,
           0.17092867, 0.17716535, 0.15555556])

```python
conv_f_4
```

    array([2.94117647e-02, 8.66551126e-04, 7.50951802e-07])

Para que exista convergencia el cociente debe ser menor a $1$.

```python
plt.figure(figsize=(12, 6))
plt.plot(conv_b, 'bs', label="Bisección")
plt.plot(conv_f_3, 'y.', label="IPF con " + r"$g_3(x)$")
plt.plot(conv_f_4, 'go', label="IPF con " + r"$g_4(x)$")
plt.yscale('log')
plt.legend()
plt.grid(True)
plt.xlabel("# de iteraciones")
plt.ylabel(r"$\epsilon_{i+1}/\epsilon_i$")
plt.show()
```

Parece

### Tasa $S$

Si $r=g(r)$ y $S = |g'(r)| < 1$.

En el caso de *IPF*:
* $g_1(x)=\frac{2}{x} \implies g_1'(x)=-\frac{2}{x^2}$, $S=|-1|=1$
* $g_2(x)=2+x-x^2 \implies g_2'(x)=1-2x$, $S=|1-2\sqrt{2}|\approx 1.83$
* $g_3(x)=\frac{2+x^3}{x^2 + x} \implies g_3'(x)=\frac{x^4+2 x^3-4x-2}{x^2 (1 + x)^2}$, $S\approx 0.172$
* $g_4(x)=\frac1x + \frac{x}{2} \implies g_4'(x)=-\frac{1}{x^2} + \frac12$, $S=|-\frac{1}{2}+\frac12| =0$

Teóricamente hay mejores elecciones de $g(x)$, ¿se valida con los resultados experimentales?

# Otro problema

Resolver:

\begin{equation}
    f(x)=x^2-x-1$
\end{equation}

## Bisección

Simplemente aplicar el algoritmo en el intervalo adecuado.

```python
f = lambda x: x ** 2 - x - 1
```

```python
x = np.linspace(-2, 2, 100)
```

```python
plt.figure(figsize=(12, 6))
plt.plot(x, f(x))
plt.grid(True)
plt.xlabel(r"$x$")
plt.ylabel(r"$f(x)$")
plt.show()
```

Como existen $2$ raíces, hay que seleccionar el intervalo según la raíz que necesitemos. Para $x_1$ se utilizará $[-2,0]$ y $[0,2]$ para $x_2$.

```python
x_b_1 = bisect(f, -2, 0)
x_b_2 = bisect(f, 0, 2)
```

```python
x_b_1[-1], x_b_2[-1]
```

    (-0.6180339902639389, 1.618033990263939)

## Solución analítica:

\begin{equation}
    x_1 = \frac{1-\sqrt{5}}{2}, \quad x_2 = \frac{1+\sqrt{5}}{2}
\end{equation}

### Error de las aproximaciones

```python
n_i_1 = np.arange(1, len(x_b_1) + 1)
n_i_2 = np.arange(1, len(x_b_2) + 1)
er_1 = error(x_b_1, (1 - np.sqrt(5)) / 2)
er_2 = error(x_b_2, (1 + np.sqrt(5)) / 2)
plt.figure(figsize=(12, 6))
plt.plot(n_i_1, er_1, 'b.')
plt.plot(n_i_2, er_2, 'r.')
plt.grid(True)
plt.xlabel("# iteraciones")
plt.ylabel("Error")
plt.yscale('log')
plt.show()
```

Como era de esperarse con *bisección*, este decae lineal con respecto al número de iteraciones.

## Iteración de Punto Fijo

### Definición de funciones $g(x)$

1. $f(x)=0$
2. $x = g(x) \implies f(x)=x-g(x)$

#### Versión 1
\begin{equation}
    \begin{split}
        x & = x^2-1 \\
        g_1(x) & = x^2-1
    \end{split}
\end{equation}

#### Versión 2
\begin{equation}
    \begin{split}
        x^2 &= x + 1 \quad / \frac1x \\
        x &= 1 + \frac1x \\
        g_2(x) &= 1 + \frac1x
    \end{split}
\end{equation}

#### Versión 3
\begin{equation}
    \begin{split}
        x^2-x &= 1 \\
        x(x-1) &= 1 \\
        x &= \frac{1}{x-1} \\
        g_3(x) &= \frac{1}{x-1}
    \end{split}
\end{equation}

#### Versión 4
\begin{equation}
    \begin{split}
        x^2 &= x + 1  \quad +x^2\\
        2x^2 &= x^2 +x + 1 \quad \frac{1}{2x} \\
        x &= \frac{x}{2} + \frac{1}{2x} + \frac12 \\
        g_4(x) &= \frac{x}{2} + \frac{1}{2x} + \frac12
    \end{split}
\end{equation}

### ¿Cuál nos servirá?

* $S_1=|g_1'(x)|=|2x|$
* $S_2=|g_2'(x)|=\left|-\frac{1}{x^2}\right|$
* $S_3=|g_3'(x)|=\left|-\frac{1}{(x-1)^2}\right|$
* $S_4=|g_4'(x)|=\left|\frac12 - \frac{1}{2x^2}\right|$

```python
g1 = lambda x: x ** 2 - 1
g2 = lambda x: 1 + 1 / x
g3 = lambda x: 1 / (x - 1)
g4 = lambda x: 1/2 *(x + 1/x + 1)
g1p = lambda x: np.abs(2 * x)
g2p = lambda x: np.abs(- 1 / x ** 2)
g3p = lambda x: np.abs(-1 / (x - 1) ** 2)
g4p = lambda x: np.abs(1 / 2 * (1 - 1 / x**2))
```

Se puede realizar un análisis de forma visual, pero hay que evitar los problemas de indeterminación de las funciones $g'(x)$.

```python
# x para evitar indeterminación...
xa = np.linspace(-1+1e-8, -1e-8)
xb = np.linspace(1e-8, 1-1e-8)
xc = np.linspace(1+1e-8, 2)
```

```python
plt.figure(figsize=(12, 6))
plt.plot(xa, g1p(xa), 'b-', label=r"$|g_1'(x)|$")
plt.plot(xb, g1p(xb), 'b-')
plt.plot(xa, g2p(xa), 'r-', label=r"$|g_2'(x)|$")
plt.plot(xb, g2p(xb), 'r-')
plt.plot(xc, g2p(xc), 'r-')
plt.plot(xa, g3p(xa), 'g-', label=r"$|g_3'(x)|$")
plt.plot(xb, g3p(xb), 'g-')
plt.plot(xc, g3p(xc), 'g-')
plt.plot(xa, g4p(xa), 'k-', label=r"$|g_4'(x)|$")
plt.plot(xb, g4p(xb), 'k-')
plt.plot(xc, g4p(xc), 'k-')
plt.axvline(x=(1-np.sqrt(5))/2, color='m', linestyle='--')
plt.axvline(x=(1+np.sqrt(5))/2, color='m', linestyle='--')
plt.ylim([0., 1.1])
plt.xlim([-1, 2])
plt.legend()
plt.grid(True)
plt.show()
```

Otra alternativa es analizar el valor de $|g'(x)|$ en $r$. Si no se conoce $r$ podemos buscar en un vecindario, por ejemplo $(r-\varepsilon, r+\varepsilon)$. $\varepsilon$ debe ser lo suficientemente pequeño, ya que segun las características de $g'(x)$ se puede concluir algo completamente erróneo.

```python
g1p(-0.6), g1p(3/2)
```

    (1.2, 3.0)

Se descarta $g_1(x)$

```python
g2p(-0.6), g2p(3/2)
```

    (2.7777777777777777, 0.4444444444444444)

$g_2(x)$ nos sirve para la segunda raíz.

```python
g3p(-0.6), g3p(3/2)
```

    (0.39062499999999994, 4.0)

$g_3(x)$ nos sirve para la primera raíz.

```python
g4p(-0.6), g4p(3/2)
```

    (0.8888888888888888, 0.2777777777777778)

$g_4(x)$ nos sirve para ambas raíces, pero para la primera converge un poco más lento.

### Experimentos

Recordar que debemos escoger $x_0$ cercano a la raíz que necesitamos.

$g_2(x)$

```python
x_2 = fpi(g2, 1, 20)
x_2
```

    array([2.        , 1.5       , 1.66666667, 1.6       , 1.625     ,
           1.61538462, 1.61904762, 1.61764706, 1.61818182, 1.61797753,
           1.61805556, 1.61802575, 1.61803714, 1.61803279, 1.61803445,
           1.61803381, 1.61803406, 1.61803396, 1.618034  , 1.61803399])

$g_3(x)$

```python
x_1 = fpi(g3, -1, 20)
x_1
```

    array([-0.5       , -0.66666667, -0.6       , -0.625     , -0.61538462,
           -0.61904762, -0.61764706, -0.61818182, -0.61797753, -0.61805556,
           -0.61802575, -0.61803714, -0.61803279, -0.61803445, -0.61803381,
           -0.61803406, -0.61803396, -0.618034  , -0.61803399, -0.61803399])

```python
conv_x_1 = conv(x_1)
conv_x_2 = conv(x_2)
```

```python
plt.figure(figsize=(12, 6))
plt.plot(conv_x_1)
plt.plot(conv_x_2)
plt.ylim([0.36, 0.4])
plt.xlabel("# de iteraciones")
plt.ylabel("S")
plt.grid(True)
plt.show()
```

Evaluando en $r$...

```python
g3p((1-np.sqrt(5))/2), g2p((1+np.sqrt(5))/2)
```

    (0.38196601125010515, 0.38196601125010515)

$g_4(x)$

```python
xx_1 = fpi(g4, -1, 20)
xx_1
```

    array([-0.5       , -0.75      , -0.54166667, -0.69391026, -0.5675094 ,
           -0.66479737, -0.58450755, -0.64767471, -0.59582982, -0.63708069,
           -0.60337028, -0.63036367, -0.60837475, -0.62604923, -0.61168385,
           -0.62325765, -0.61386527, -0.62144361, -0.61530003, -0.62026171])

```python
xx_2 = fpi(g4, 1, 20)
xx_2
```

    array([1.5       , 1.58333333, 1.60745614, 1.61477855, 1.61702926,
           1.61772363, 1.61793809, 1.61800436, 1.61802483, 1.61803116,
           1.61803311, 1.61803372, 1.61803391, 1.61803396, 1.61803398,
           1.61803399, 1.61803399, 1.61803399, 1.61803399, 1.61803399])

```python
conv_xx_1 = conv(xx_1)
conv_xx_2 = conv(xx_2)
```

```python
plt.figure(figsize=(12, 6))
plt.plot(conv_xx_1)
plt.plot(conv_xx_2)
plt.xlabel("# de iteraciones")
plt.ylabel("S")
plt.grid(True)
plt.show()
```

```python
g4p((1-np.sqrt(5))/2), g4p((1+np.sqrt(5))/2)
```

    (0.8090169943749472, 0.30901699437494745)

## Conclusión

Según el $g(x)$ seleccionado vamos a obtener o no convergencia para distintos $x_0$. Incluso, para una buena elección de $g(x)$ podemos encontrar más de una raíz.


# Revisão de conceitos estatísticos III (Estimadores)

Vamos explorar alguns conceitos estatísticos aplicados à análise de sinais.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
from scipy.stats import norm
plt.rcParams.update({'font.size': 14})
import IPython.display as ipd # to play signals
import sounddevice as sd
```

# Um conjunto de ruídos brancos

Vamos gerar um conjunto (ensemble) de ruídos brancos. Você pode encarar isso como se fosse uma série de gravações de um sinal aleatório. Por exemplo, como se você fosse medir a vibração em uma máquina complexa e tomasse $N_{rec}$ gravações. No caso do ruído branco temos um fenômeno governado por uma distribuição de probabilidade Gaussiana e constante com o tempo:

\begin{equation}
p(x) = \mathcal{N}(\mu_x, \sigma_x) = \frac{1}{\sqrt{2\pi}\sigma_x}\mathrm{e}^{-\frac{1}{2\sigma_x^2}(x-\mu_x)^2}
\end{equation}
em que $\mu_x$ é a média e $\sigma_{x}$ é o desvio padrão.

```python
# parâmetros da distribuição de probabilidade do fenômeno
mu_x = 0.0    # média
sigma_x = 0.5 # desvio padrão - var = 0.25 (RMS**2)

# A densidade de probabilidade
x = np.arange(-10, 10, 0.01)
px = norm.pdf(x, loc= mu_x, scale = sigma_x)

# Número de gravações
N_rec = 2000

# Os sinais - vão ser gravados numa matriz N_rec x len(time)
fs = 2000
time = np.arange(0,2, 1/fs)
xt = np.random.normal(loc = mu_x, scale = sigma_x, size=(N_rec,len(time)))
```

# Vamos escolher 5 gravações do conjunto de sinais gravados para plotar $x(t)$

```python
rec_choose = np.random.randint(N_rec+1, size=5)
color = ['b', 'r', 'k', 'g', 'magenta']

fig, axs = plt.subplots(5, 1, figsize = (10, 8))
for i in np.arange(5):
    axs[i].plot(time, xt[rec_choose[i]], linewidth = 1, color = color[i])
    axs[i].axvline(0.25, color='grey',linestyle = '--', linewidth = 4, alpha = 0.8)
    axs[i].set_ylabel(r'$x(t)$')
    axs[i].set_xlim((0, time[-1]))
    axs[i].set_ylim((-4, 4))
axs[i].set_xlabel('tempo [s]')
plt.tight_layout()
```

# Computar os momentos para o estimador do conjunto.

```python
# Escolha 1 ou 2 instantes de tempo e calcule as amostras
n_1 = int(0.25*fs)
n_2 = int(0.73*fs)

# Primeiro momento
Ex_1 = (1/N_rec)*np.sum(xt[:, n_1])
Ex_2 = (1/N_rec)*np.sum(xt[:, n_2])
print("E[x] para t_1 é {:.4f}".format(Ex_1))
print("E[x] para t_2 é {:.4f}".format(Ex_2))

# Segundo momento
Ex2_1 = (1/N_rec)*np.sum((xt[:, n_1]-Ex_1)**2)
Ex2_2 = (1/N_rec)*np.sum((xt[:, n_2]-Ex_2)**2)
print("E[(x-\mu_x)^2] para t_1 é {:.4f}".format(Ex2_1))
print("E[(x-\mu_x)^2] para t_2 é {:.4f}".format(Ex2_2))
```

    E[x] para t_1 é 0.0157
    E[x] para t_2 é -0.0568
    E[(x-\mu_x)^2] para t_1 é 0.3184
    E[(x-\mu_x)^2] para t_2 é 0.3158

# Agora, tome apenas 1 das gravações (estimador do sinal)

Encare isso como sendo a única gravação que você fez.

```python
rec_choose = np.random.randint(N_rec+1, size=1)
xt_rec = np.reshape(xt[rec_choose,:], len(time))

plt.figure(figsize = (10, 3))
plt.plot(time, xt_rec, linewidth = 1, color = 'g')
plt.ylabel(r'$x(t)$')
plt.xlim((0, time[-1]))
plt.ylim((-4, 4))
plt.xlabel('tempo [s]')
plt.tight_layout()
```

# Computar os momentos do sinal.

```python
# Primeiro momento
Ex = np.mean(xt_rec)
Ex2 = np.std(xt_rec)**2
x_rms = np.sqrt((1/len(time))*np.sum(xt_rec**2))

print("E[x] para o sinal é {:.4f}".format(Ex))
print("E[(x-\mu_x)^2] para o sinal é {:.4f}".format(Ex2))
print("O quadrado do valor RMS é {:.4f}".format(x_rms**2))
```

    E[x] para o sinal é -0.0010
    E[(x-\mu_x)^2] para o sinal é 0.2436
    O quadrado do valor RMS é 0.2436

# Computar a distribuição dos momentos calculados

```python
# Estimador do conjunto
mu_x_conj = np.mean(xt, axis = 0)
std_x_conj = np.std(xt, axis = 0)

# Figura
fig, axs = plt.subplots(1, 2, figsize = (12, 4))
axs[0].hist(mu_x_conj, bins = np.linspace(-0.3, 0.3, 50))
axs[0].grid(linestyle = '--', which='both')
axs[0].set_ylabel('Contagem [-]')
axs[0].set_xlabel(r'$\mu_x$ [-]')

axs[1].hist(std_x_conj, bins = np.linspace(0.3, 0.7, 50))
axs[1].grid(linestyle = '--', which='both')
axs[1].set_ylabel('Contagem [-]')
axs[1].set_xlabel(r'$\sigma_x$ [-]')
plt.tight_layout()
```

```python
# Estimador do sinal
mu_x_rec = np.mean(xt, axis = 1)
std_x_rec = np.std(xt, axis = 1)

# Figura
fig, axs = plt.subplots(1, 2, figsize = (12, 4))
axs[0].hist(mu_x_rec, bins = np.linspace(-0.3, 0.3, 50))
axs[0].grid(linestyle = '--', which='both')
axs[0].set_ylabel('Contagem [-]')
axs[0].set_xlabel(r'$\mu_x$ [-]')

axs[1].hist(std_x_rec, bins = np.linspace(0.3, 0.7, 50))
axs[1].grid(linestyle = '--', which='both')
axs[1].set_ylabel('Contagem [-]')
axs[1].set_xlabel(r'$\sigma_x$ [-]')
plt.tight_layout()
```

Aula 51  - Rev conceitos estatisticos III/conceitos estatisticos III.ipynb


# 10 Modeling a Ball Channel Pendulum

Below is a video of a simple cardboard pendulum that has a metal ball in a semi-circular channel mounted above the pendulum's rotational joint. It is an interesting dynamic system that can be constructed and experimented with. This system seems to behave like a single degree of freedom system, i.e. that the ball's location is a kinematic function of the pendulum's angle. But this may not actually be the case. It depends on the nature of the motion of the ball with respect to the channel. If the ball rolls without slipping in the channel it is a single degree of freedom system. If the ball can slip and roll it is at a minimum a two degree of freedom system. In this notebook we will derive the equations of motion of the system considering the ball slips and doesn't roll in the channel.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('3pJdkssUdfU', width=600, height=480)
```

# Imports and setup

```python
import sympy as sm
import numpy as np
```

```python
sm.init_printing()
```

```python
%matplotlib notebook
```

# Free Body Diagram

TODO : Add the free body diagram we drew in class.

# Constants

Create a symbol for each of the system's contant parameters.

- $m_p$: mass of the pendulum
- $m_b$: mass of the ball
- $l$: length of the pendulum
- $r$: radius of the channel
- $g$: acceleration due to gravity

```python
mp, mb, r, l, g = sm.symbols('m_p, m_b, r, l, g')
```

# Generalized Coordinates

Create functions of time for each generalized coordinate.

- $\theta(t)$: angle of the pendulum
- $\phi(t)$: angle of the line from the center of the channel to the ball

```python
t = sm.symbols('t')
```

```python
theta = sm.Function('theta')(t)
phi = sm.Function('phi')(t)
```

```python
theta.diff()
```

```python
theta.diff(t)
```

# Kinetic Energy

Write the kinetic energy in terms of the generalized coordinates.

```python
Tp = mp * (l * theta.diff())**2 / 2
Tp
```

```python
Tb = mb / 2 * ((-r * theta.diff() * sm.cos(theta) + phi.diff() * r * sm.cos(theta + phi))**2 +
               (-r * theta.diff() * sm.sin(theta) + phi.diff() * r * sm.sin(theta + phi))**2)
Tb
```

```python
T = Tp + Tb
T
```

# Potential Energy

Each particle (pendulum bob and the ball) has a potential energy associated with how high the mass rises.

```python
U = mp * g * (l - l * sm.cos(theta)) + mb * g * (l + r * sm.cos(theta) - r * sm.cos(theta + phi))
U
```

# Lagrange's equation of the second kind

There are two generalized coordinates with two degrees of freedom and thus two equations of motion.

$$
0 = f_\theta(\theta, \phi, \dot{\theta}, \dot{\phi}, \ddot{\theta}, \ddot{\phi}, t) \\
0 = f_\phi(\theta, \phi, \dot{\theta}, \dot{\phi}, \ddot{\theta}, \ddot{\phi}, t) \\
$$

```python
L = T - U
L
```

```python
f_theta = L.diff(theta.diff()).diff(t) - L.diff(theta)
f_theta = sm.trigsimp(f_theta)
f_theta
```

```python
f_phi = L.diff(phi.diff()).diff(t) - L.diff(phi)
f_phi = sm.trigsimp(f_phi)
f_phi
```

# Explicit first order form

Introduce two new variables for the generalized speeds:

$$
\alpha = \dot{\theta} \\
\beta = \dot{\phi}
$$

```python
alpha = sm.Function('alpha')(t)
beta = sm.Function('beta')(t)
```

Substitute these into the two equations of motion:

```python
f_theta = f_theta.subs({theta.diff(): alpha, phi.diff(): beta})
f_theta
```

```python
f_phi = f_phi.subs({theta.diff(): alpha, phi.diff(): beta})
f_phi
```

```python
f = sm.Matrix([f_theta, f_phi])
f
```

The equations are motion are based on Newton's second law and Euler's equations, thus it is guaranteed that terms in $\mathbf{f}$ that include $\dot{\mathbf{u}}$ are linear with respect to $\dot{\mathbf{u}}$. So the equations of motion can be written in this matrix form:

$$
\mathbf{f}(\mathbf{q}, \mathbf{u}, \dot{\mathbf{u}}, t) = \mathbf{M}(\mathbf{q}, t)\dot{\mathbf{u}} - \mathbf{G}(\mathbf{q}, \mathbf{u}, t) = 0
$$

$\mathbf{M}$ is called the "mass matrix" of the nonlinear equations. If the derivatives of $\mathbf{f}$ with respect to $\dot{\mathbf{u}}$ are computed, i.e. the [Jacobian](https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant) of $\mathbf{f}$ with respect to $\dot{\mathbf{u}}$, then you can obtain the mass matrix.

```python
u = sm.Matrix([alpha, beta])
u
```

```python
M = f.jacobian(u.diff())
M
```

$$\mathbf{G} = \mathbf{M}\dot{\mathbf{u}} - \mathbf{f}$$

```python
# TODO : The `sm.MutableDenseMatrix()` call is due to a regression present in SymPy 1.2 and 1.3.
# Once that is fixed, this should not be necessary.
G = M * sm.MutableDenseMatrix(u.diff()) - f
G.simplify()
G
```

The first order form has all of the $\dot{\mathbf{u}}$ on the left hand side. This requires solving the linear system of equations:

$$\mathbf{M}\dot{\mathbf{u}}=\mathbf{G}$$

The mathematical solution is:

$$\dot{\mathbf{u}}=\mathbf{M}^{-1}\mathbf{G}$$

```python
udot = M.inv() * G
udot.simplify()
udot
```

A better way to solve the system of linear equations is to use Guassian elmination. SymPy has a variety of methods for sovling linear systems. The LU decomposition method of Guassian elimination is a generally good choice for this and for large number of degrees of freedom this will provide reasonable computation time. For very large $n$ this should be done numerically instead of symbolically.

```python
udot = M.LUsolve(G)
udot.simplify()
udot
```

Note the differences in timing below. For systems with a large number of degrees of freedom, this gap in timing will increase significantly.

```python
%timeit M.inv() * G
```

```python
%timeit M.LUsolve(G)
```

# Simulation of the nonlinear system

Resonance has a prepared system that is only missing the equations of motion.

```python
from resonance.nonlinear_systems import BallChannelPendulumSystem
```

```python
sys = BallChannelPendulumSystem()
```

```python
sys.constants
```

```python
sys.coordinates
```

```python
sys.speeds
```

The full first order ordinary differential equations are:

$$
\dot{\theta} = \alpha \\
\dot{\phi} = \beta \\
\dot{\alpha} = f_{\alpha}(\theta, \phi, \alpha, \beta, t) \\
\dot{\beta} = f_{\beta}(\theta, \phi, \alpha, \beta, t)
$$

where:

$$
\dot{\mathbf{q}}=\begin{bmatrix}
\theta \\
\phi
\end{bmatrix} \\
\dot{\mathbf{u}}=\mathbf{M}^{-1}\mathbf{G} =
\begin{bmatrix}
f_{\alpha}(\theta, \phi, \alpha, \beta, t) \\
f_{\beta}(\theta, \phi, \alpha, \beta, t)
\end{bmatrix}
$$

Introducing:

$$\mathbf{x} =
\begin{bmatrix}
\mathbf{q} \\
\mathbf{u}
\end{bmatrix}=
\begin{bmatrix}
\theta \\
\phi \\
\alpha \\
\beta
\end{bmatrix}
$$

we have equations for:

$$\dot{\mathbf{x}} = \begin{bmatrix}
\dot{\theta} \\
\dot{\phi} \\
\dot{\alpha} \\
\dot{\beta}
\end{bmatrix}
=
\begin{bmatrix}
\alpha \\
\beta \\
f_{\alpha}(\theta, \phi, \alpha, \beta, t) \\
f_{\beta}(\theta, \phi, \alpha, \beta, t)
\end{bmatrix}
$$

To find $\mathbf{x}$ we must integrate $\dot{\mathbf{x}}$ with respect to time:

$$
\mathbf{x} = \int_{t_0}^{t_f} \dot{\mathbf{x}} dt
$$

Resonance uses numerical integration behind the scenes to compute this integral. Numerical integration routines typicall require that you write a function that computes the right hand side of the first order form of the differential equations. This function takes the current state and time and computes the derivative of the states.

SymPy's `lambdify` function can convert symbolic expression into NumPy aware functions, i.e. Python functions that can accept NumPy arrays.

```python
eval_alphadot = sm.lambdify((phi, theta, alpha, beta, mp, mb, l, r, g), udot[0])
```

```python
eval_alphadot(1, 2, 3, 4, 5, 6, 7, 8, 9)
```

```python
eval_betadot = sm.lambdify((phi, theta, alpha, beta, mp, mb, l, r, g), udot[1])
```

```python
eval_betadot(1, 2, 3, 4, 5, 6, 7, 8, 9)
```

Now the right hand side function can be written:

```python
def rhs(phi, theta, alpha, beta, mp, mb, l, r, g):
    theta_dot = alpha
    phi_dot = beta
    alpha_dot = eval_alphadot(phi, theta, alpha, beta, mp, mb, l, r, g)
    beta_dot = eval_betadot(phi, theta, alpha, beta, mp, mb, l, r, g)
    return theta_dot, phi_dot, alpha_dot, beta_dot
```

```python
rhs(1, 2, 3, 4, 5, 6, 7, 8, 9)
```

This function also works with numpy arrays:

```python
rhs(np.array([1, 2]), np.array([3, 4]), np.array([5, 6]), np.array([7, 8]), 9, 10, 11, 12, 13)
```

Add this function as the differential equation function of the system.

```python
sys.diff_eq_func = rhs
```

Now the `free_response` function can be called to simulation the nonlinear system.

```python
traj = sys.free_response(20, sample_rate=500)
```

```python
traj[['theta', 'phi']].plot(subplots=True);
```

```python
sys.animate_configuration(fps=30)
```

# Equilibrium

This system four equilibrium points.

1. $\theta = \phi = \alpha = \beta = 0$
2. $\theta = \pi, \phi=\alpha=\beta=0$
3. $\theta = \pi, \phi = -\pi, \alpha=\beta=0$
4. $\theta = 0, \phi = \pi, \alpha=\beta=0$

If you set the velocities and accelerations equal to zero in the equations of motion you can then solve for the coordinates that make these equations equal to zero. This is the static force balance equations.

```python
f_static = f.subs(dict(zip(u.diff(), [0, 0]))).subs(dict(zip(u, [0, 0])))
f_static
```

```python
sm.solve(f_static, theta, phi)
```

Let's look at the simulation with the initial condition very close to an equilibrium point:

```python
sys.coordinates['theta'] = np.deg2rad(180.0000001)
sys.coordinates['phi'] = -np.deg2rad(180.00000001)
```

```python
traj = sys.free_response(20, sample_rate=500)
```

```python
sys.animate_configuration(fps=30)
```

```python
traj[['theta', 'phi']].plot(subplots=True);
```

This equlibrium point is an *unstable equlibrium*.

# Linearizing the system

The equations of motion can be linearized about one of the equilibrium points. This can be done by computing the linear terms of the multivariate Taylor Series expansion. This expansion can be expressed as:

$$
\mathbf{f}_{linear} = \mathbf{f}(\mathbf{v}_{eq}) + \mathbf{J}_f(\mathbf{v}_{eq}) (\mathbf{v} - \mathbf{v}_{eq})
$$

where $\mathbf{J}_f$ is the Jacobian of $\mathbf{f}$ with respect to $\mathbf{v}$ and:

$$
\mathbf{v} = \begin{bmatrix}
\theta\\
\phi \\
\alpha \\
\beta \\
\dot{\alpha} \\
\dot{\beta}
\end{bmatrix}
$$

In our case let's linearize about the static position where $\theta=\phi=0$.

```python
f
```

```python
v = sm.Matrix([theta, phi, alpha, beta, alpha.diff(), beta.diff()])
v
```

```python
v0 = sm.zeros(len(v), 1)
v0
```

```python
v_eq_sub = dict(zip(v, v0))
v_eq_sub
```

The linear equations are then:

```python
f_lin = f.subs(v_eq_sub) + f.jacobian(v).subs(v_eq_sub) * (v - v0)
f_lin
```

Note that all of the terms that involve the coordinates, speeds, and their derivatives are linear terms, i.e. simple linear coefficients. These linear equations can be put into this canonical form:

$$\mathbf{M}\dot{\mathbf{u}} + \mathbf{C}\mathbf{u} + \mathbf{K} \mathbf{q} = \mathbf{F}$$

with:

- $\mathbf{M}$ as the mass matrix
- $\mathbf{C}$ as the damping matrix
- $\mathbf{K}$ as the stiffness matrix
- $\mathbf{F}$ as the forcing vector

The Jacobian can again be utlized to extract the linear coefficients.

```python
q = sm.Matrix([theta, phi])
q
```

```python
u
```

```python
M = f_lin.jacobian(u.diff())
M
```

```python
C = f_lin.jacobian(u)
C
```

```python
K = f_lin.jacobian(q)
K
```

```python
F = sm.simplify(f_lin - M * u.diff() - C * u - K * q)
F
```

# Simulate the linear system

```python
from resonance.linear_systems import BallChannelPendulumSystem
```

```python
lin_sys = BallChannelPendulumSystem()
```

For linear systems, a function that calculates the canonical coefficient matrices should be created. Each of the canonical matrices should be created as 2 x 2 NumPy arrays.

```python
def canon_coeff_matrices(mp, mb, l, g, r):
    M = np.array([[mp * l**2 + mb * r**2, -mb * r**2],
                  [-mb * r**2, mb * r**2]])
    C = np.zeros((2, 2))
    K = np.array([[g * l * mp, g * mb * r],
                  [g * mb * r, g * mb * r]])
    return M, C, K
```

```python
lin_sys.canonical_coeffs_func = canon_coeff_matrices
```

```python
M_num, C_num, K_num = lin_sys.canonical_coefficients()
```

```python
M_num
```

```python
C_num
```

```python
K_num
```

```python
lin_traj = lin_sys.free_response(20, sample_rate=500)
```

```python
lin_traj[['theta', 'phi']].plot(subplots=True);
```

# Compare the nonlinear and linear simulations

```python
sys.coordinates['theta'] = np.deg2rad(10)
sys.coordinates['phi'] = np.deg2rad(-10)
```

```python
lin_sys.coordinates['theta'] = np.deg2rad(10)
lin_sys.coordinates['phi'] = np.deg2rad(-10)
```

```python
traj = sys.free_response(10.0)
```

```python
lin_traj = lin_sys.free_response(10.0)
```

```python
axes = traj[['theta', 'phi']].plot(subplots=True, color='red')
axes = lin_traj[['theta', 'phi']].plot(subplots=True, color='blue', ax=axes)
axes[0].legend([r'nonlin $\theta$', r'lin $\theta$'])
axes[1].legend([r'nonlin $\phi$', r'lin $\phi$']);
```


# Regularization

## 1. Introduction

### 1.1 Libraries

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model
from scipy.optimize import minimize
from numpy.random import permutation
from sympy import var, diff, exp, latex, factor, log, simplify
from IPython.display import display, Math, Latex
np.set_printoptions(precision=4)
%matplotlib inline
```

### 1.2 Dataset

The in-sample and out-of-sample data is retrieved from the following sources:

```bash
curl -O http://work.caltech.edu/data/in.dta
curl -O http://work.caltech.edu/data/out.dta
```

## 2. Non-linear Transforms

### 2.1 Hypothesis Set

Consider the linear transform $z_i = \phi_i\left(\mathbf{x}\right)$ or $\mathbf{z} = \Phi\left(\mathbf{x}\right)$, with the following mapping:

$$\mathbf{x} = \left(x_0, x_1, \dots, x_d\right) \rightarrow \mathbf{z} = \left(z_0, z_1, \dots, z_{\tilde d}\right)$$

The final hypothesis, $\mathcal{X}$ space is:

$$g\left(\mathbf{x}\right) = \mathbf{\tilde w^T} \Phi\left(\mathbf{x}\right)$$

$$g\left(\mathbf{x}\right) = \left(w_0, w_1, w_2, w_3, w_4, w_5, w_6, w_7\right) \left(\begin{array}{c}1\\x_1\\x_2\\x_1^2\\x_2^2\\x_1 x_2\\|x_1 - x_2|\\|x_1 + x_2|\end{array}\right) = w_0 + w_1 x_1 + w_2 x_2 + \dots$$

To implement the above hypothesis set on an arbitrary matrix $\mathbf{X}$,
we augment the matrix $\mathbf{X}$ with a *ones column*, and
non-linear features are added as new columns via the following two subroutines.

```python
def add_ones_column(X):
    n = X.shape[0]
    x0 = np.ones((n,1))
    return np.hstack((x0,X))
```

```python
def add_nonlinear_features(X,n):
    """
    Here 'n' can be 3 to 7, corresponding to one of the following ϕ_n:
    ϕ_0 = 1
    ϕ_1 = 1 + x_1
    ϕ_2 = 1 + x_1 + x_2
    ϕ_3 = 1 + x_1 + x_2 + x_1^2
    ϕ_4 = 1 + x_1 + x_2 + x_1^2 + x_2^2
    ϕ_5 = 1 + x_1 + x_2 + x_1^2 + x_2^2 + x_1 x_2
    ϕ_6 = 1 + x_1 + x_2 + x_1^2 + x_2^2 + x_1 x_2 + |x_1 - x_2|
    ϕ_7 = 1 + x_1 + x_2 + x_1^2 + x_2^2 + x_1 x_2 + |x_1 - x_2| + |x_1 + x_2|
    """
    m = X.shape[0]
    X = np.hstack((X,np.zeros((m,n-X.shape[1]+1))))
    if n >= 3: X[:,3] = X[:,1]**2
    if n >= 4: X[:,4] = X[:,2]**2
    if n >= 5: X[:,5] = X[:,1]*X[:,2]
    if n >= 6: X[:,6] = np.abs(X[:,1] - X[:,2])
    if n >= 7: X[:,7] = np.abs(X[:,1] + X[:,2])
    return(X)
```

### 2.2 Loading and Transforming the Data

The data is loaded and transformed as follows:

```python
training = np.loadtxt("data/in.dta")
X_train = add_ones_column(training[:,:-1])
y_train = training[:,-1]
X_train = add_nonlinear_features(X_train,7)
n_train = y_train.shape[0]

testing  = np.loadtxt("data/out.dta")
X_test  = add_ones_column(testing[:,:-1])
y_test  = testing[:,-1]
X_test  = add_nonlinear_features(X_test,7)
n_test = y_test.shape[0]

n_features = X_train.shape[1]
```

### 2.2 Linear Regression (without regularization)

The [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) (OLS) estimator for the weights is:

$$w = \left(\mathbf{X^T X}\right)^{-1}\mathbf{X^T y} = \mathbf{X^\dagger}y$$

where $\mathbf{X^\dagger} = \left(\mathbf{X^T X}\right)^{-1}\mathbf{X^T}$ is the [Moore–Penrose pseudoinverse](https://en.wikipedia.org/wiki/Moore–Penrose_pseudoinverse).

This is implemented via the following line of code in the subroutine below:

`w_linreg = np.dot(np.linalg.pinv(X),y)`

```python
def train_without_regularization(X_train,y_train):
    w_no_reg = np.dot(np.linalg.pinv(X_train),y_train)
    return w_no_reg

def get_E_in_E_out(X_test,y_test,X_train,y_train,w,verbose=True):
    n_train = y_train.shape[0]
    n_test  = y_test.shape[0]
    y_train_eval = np.sign(np.dot(X_train,w)).astype(int)
    y_test_eval  = np.sign(np.dot(X_test,w)).astype(int)
    E_in = np.sum(y_train != y_train_eval) / n_train
    E_out = np.sum(y_test != y_test_eval) / n_test
    if verbose is True:
        print("E_in = {}".format(E_in))
        print("E_out = {}".format(E_out))
    return E_in,E_out
```

The un-regularized in-sample and out-of-sample error can be calculated as follows (in the next section, the regularized in-sample and out-of-sample errors will be calculated):

```python
w_linreg = np.dot(np.linalg.pinv(X_train),y_train)
y_train_linreg = np.sign(np.dot(X_train,w_linreg))
y_test_linreg  = np.sign(np.dot(X_test,w_linreg))
E_in = np.sum(y_train != y_train_linreg) / n_train
E_out = np.sum(y_test != y_test_linreg) / n_test
print("E_in = {}".format(E_in))
print("E_out = {}".format(E_out))
```

    E_in = 0.02857142857142857
    E_out = 0.084

### 2.3 Linear Regression (with Regularization)

#### 2.3.1 Background

In order to perform regularization, a "soft-order" constraint is employed for the weights:

$$\sum\limits_{q=0}^Q w_q^2 \le C$$

Subject to the above constraint, our aim is to minimize the following cost function on the in-sample points:

$$E_{in}\left(\mathbf{w}\right) = \frac{1}{N} \sum\limits_{n=1}^N \left(\mathbf{w^T z_n - y_n}\right)^2 = \frac{1}{N}\left(\mathbf{Zw-y}\right)^T\left(\mathbf{Zw-y}\right)$$

The minimum of the above function occurs when:

$$\nabla E_{in}\left(\mathbf{w}\right) \propto -\mathbf{w_{reg}}$$
$$\nabla E_{in}\left(\mathbf{w}\right) = -2\frac{\lambda}{N}\mathbf{w_{reg}}$$

where in the last step, we have introduced $2\frac{\lambda}{N}$ as our constant of proportionality.

Hence, we wish to solve:

$$\nabla E_{in}\left(\mathbf{w_{reg}}\right) + 2\frac{\lambda}{N}\mathbf{w_{reg}} = 0$$

Notice that solving the above equation corresponds to minimizing the following:

$$E_{in}\left(\mathbf{w}\right) + \frac{\lambda}{N}\mathbf{w^T w}$$

This new cost function can be viewed as an augmented error, which we can write as follows:

$$E_{aug}\left(\mathbf{w}\right) = \frac{1}{N}\left(\mathbf{Zw-y}\right)^T \left(\mathbf{Zw-y}\right) + \frac{\lambda}{N}\mathbf{w^T w}$$

Minimizing the above cost function is equivalent to minimizing $E_{in}\left(\mathbf{w}\right)$ subject to the constraint $\mathbf{w^T w} \le C$

To determine the **regularized weights**, we minimize the augmented error as follows:

$$\nabla E_{aug}\left(\mathbf{w}\right) = 0$$

$$\mathbf{Z^T} \left(\mathbf{Zw-y}\right) + \lambda\mathbf{w} = 0$$

$$\left(\mathbf{Z^T Z} + \lambda\mathbf{I}\right)\mathbf{w} - \mathbf{Z^T y} = 0$$

$$\mathbf{w_{reg}} = \left(\mathbf{Z^T Z} + \lambda\mathbf{I}\right)^{-1}\mathbf{Z^T y}$$

```python
def train_with_regularization(lmbda,X_train,y_train,use_sklearn=False):
    n_samples = y_train.shape[0]
    if use_sklearn is True:
        reg = linear_model.Ridge(alpha = lmbda/n_samples, fit_intercept=True)
        reg.fit(X_train,y_train)
        w_reg = reg.coef_
        w_reg[0] = reg.intercept_
    else:
        n_features=X_train.shape[1]
        ZTy = np.dot(X_train.T,y_train)
        ZTZ = np.dot(X_train.T,X_train)
        w_reg = np.dot(np.linalg.inv(ZTZ + lmbda*np.identity(n_features)),ZTy)
    return w_reg
```

```python
def get_E_in_E_out(X_test,y_test,X_train,y_train,w):
    n_train = y_train.shape[0]
    n_test = y_test.shape[0]
    y_train_w = np.sign(np.dot(X_train,w))
    y_test_w  = np.sign(np.dot(X_test,w))
    E_in = np.sum(y_train != y_train_w) / n_train
    E_out = np.sum(y_test != y_test_w) / n_test
    return E_in, E_out
```

#### 2.3.2 Checking the above subroutine.

The data can be trained by either using the subroutine `train_without_regularization()` or setting $\lambda=0$ in the subroutine `train_with_regularization()`.  The weights obtained should be the same in either case, and this is easy to check with `numpy.allclose()`.

```python
w_no_reg = train_without_regularization(X_train,y_train)
w_reg = train_with_regularization(0.0,X_train,y_train,use_sklearn=False)
assert np.allclose(w_no_reg,w_reg)
E_in,E_out = get_E_in_E_out(X_test,y_test,X_train,y_train,w_no_reg)
```

#### 2.3.3 Finding the best $\lambda$

To find the best choice of $\lambda$, the values of $E_{in}$ and $E_{out}$ are plotted over a range of values of $\lambda$.

```python
def find_best_lambda(lmbd_arr,X_train,y_train,X_test,y_test,use_sklearn=False):
    E_in_arr = []
    E_out_arr = []
    w_arr = []
    for lmbda in lmbd_arr:
        w_reg = train_with_regularization(lmbda,X_train,y_train,use_sklearn=use_sklearn)
        E_in, E_out = get_E_in_E_out(X_test,y_test,X_train,y_train,w_reg)
        E_in_arr.append(E_in)
        E_out_arr.append(E_out)
        w_arr.append(w_reg)
    return np.array(E_in_arr), np.array(E_out_arr), np.array(w_arr)
```

```python
def plot_regularization(fig,plot_id,title,lmbd_arr,E_in_reg,E_out_reg):
    ax = fig.add_subplot(plot_id)
    ax.semilogx(lmbd_arr,E_in_reg,'x-',label="$E_{in}$")
    ax.semilogx(lmbd_arr,E_out_reg,'o-',label="$E_{out}$")
    ax.set_xlabel('$\lambda$')
    ax.grid(True)
    ax.legend(loc='best',frameon=False)
    ax.set_title(title)
```

```python
lmbd_arr1 = [0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0]
fig = plt.figure(figsize=(12,5))
E_in_reg1, E_out_reg1, w_reg1 = find_best_lambda(lmbd_arr1,X_train,y_train,X_test,y_test,False)
plot_regularization(fig,121,"",lmbd_arr1,E_in_reg1,E_out_reg1)
#E_in_sk, E_out_sk, w_sk = find_best_lambda(lmbd_arr,X_train,y_train,X_test,y_test,True)
#plot_regularization(fig,122,"sklearn",lmbd_arr,E_in_sk,E_out_sk)
lmbd_arr2 = [1.0, 2.0, 5.0, 10.0, 20.0, 50.0, 100.0, 200.0, 500.0, 1000.0]
E_in_reg2, E_out_reg2, w_reg2 = find_best_lambda(lmbd_arr2,X_train,y_train,X_test,y_test,False)
plot_regularization(fig,122,"",lmbd_arr2,E_in_reg2,E_out_reg2)
```

Here we can see that when regularization turned on ($\lambda \ne 0$):

* $\lambda = 10^{-3}$ has an $E_{in}$ of about 0.03, and an $E_{out}$ of 0.08.
* $\lambda = 10^{-1}$ has the best $E_{out}$ of about 0.06.
* $\lambda = 10^{3}$ has an $E_{in}$ and $E_{out}$ of about 0.4.

### 2.4 Validation Sets

The data may also be split into validation sets for training, and the new model evaluated on the final, held-out, test set.  In the following subroutine, the training set (`X_train_all,y_train_all`) is split one more time into a new training set (`X_train,y_train`) and a validation set (`X_valid,y_valid`).  By default, the first 25 points are designated as training points (`train_pts`).

```python
def training_validation_test_split(training,testing,train_pts=25,k=7,swap=False):
    X_train_all = add_ones_column(training[:,:-1])
    y_train_all = training[:,-1]
    X_train_all = add_nonlinear_features(X_train_all,k)

    X_test  = add_ones_column(testing[:,:-1])
    y_test  = testing[:,-1]
    X_test  = add_nonlinear_features(X_test,k)

    val_pts = X_train_all.shape[0]-train_pts
    assert val_pts == y_train_all.shape[0]-train_pts
    if swap is True:
        X_train = X_train_all[val_pts:,:]
        X_valid = X_train_all[:val_pts,:]
        y_train = y_train_all[val_pts:]
        y_valid = y_train_all[:val_pts]
    else:
        X_train = X_train_all[:train_pts,:]
        X_valid = X_train_all[train_pts:,:]
        y_train = y_train_all[:train_pts]
        y_valid = y_train_all[train_pts:]

    assert X_train.shape[0]==train_pts
    assert X_valid.shape[0]==val_pts
    assert y_train.shape[0]==train_pts
    assert y_valid.shape[0]==val_pts
    n_features = X_train.shape[1]
    return X_train_all,y_train_all,X_train,y_train,X_valid,y_valid,X_test,y_test
```

In this example, the first 3,4,5,6, or 7 terms (corresponding to $\phi_3, \phi_4, \dots, \phi_7$) of the non-linear transform are used and the classification error of the validation set is evaluated.

```python
lmbd_arr = [0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05,
            0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0, 20.0, 50.0, 100.0, 500.0, 1000.0]
k_arr = np.array([3, 4, 5, 6, 7])
fig = plt.figure(figsize=(12,10))
plot_ids = [ 321, 322, 323, 324, 325 ]
for i,k in enumerate(k_arr):
    X_train_all,y_train_all,X_train,y_train,X_valid,y_valid,X_test,y_test = \
        training_validation_test_split(training,testing,k=k)
    E_in_reg, E_val_reg, w_reg_arr = find_best_lambda(lmbd_arr,X_train,y_train,X_valid,y_valid,False)
    plot_regularization(fig,plot_ids[i],"$\phi_{}$".format(k),lmbd_arr,E_in_reg,E_val_reg)
```

Here we can see that for small $\lambda$, $\phi_6$ has the lowest classification error on the validation set.

### 2.5 Plotting

The in-sample and out-of-sample error from the non-linear transforms can be visualized via contour plots.  In this case, we will use $\lambda = 0.1$ which had the lowest out-of-sample error.

```python
def plot_data_nonlinear(fig,plot_id,X,y,w_arr,w_colors,titles):
    p = 2.0
    x1 = np.linspace(-p,p,100)
    x2 = np.linspace(-p,p,100)
    X1,X2 = np.meshgrid(x1,x2)
    X1_sq= X1**2
    X2_sq= X2**2
    X1X2 = X1*X2
    X1mX2= np.abs(X1-X2)
    X1pX2= np.abs(X1+X2)

    for i,w in enumerate(w_arr):
        G = w[0] + w[1]*X1 + w[2]*X2 + w[3]*X1_sq + w[4]*X2_sq + \
            w[5]*X1X2 + w[6]*X1mX2 + w[7]*X1pX2
        ax = fig.add_subplot(plot_id[i])
        ax.plot(X[y > 0,1],X[y > 0,2],'b+',label='Positive (+)')
        ax.plot(X[y < 0,1],X[y < 0,2],'r_',label='Negative (-)')
        cp0 = ax.contour(X1,X2,G,1,linewidth=4, levels=[0.0],
                         colors=w_colors[i])
        ax.clabel(cp0, inline=True, fontsize=14)
        #cp1 = ax.contour(X1,X2,Y,N=1,linewidth=4, levels=[-1.0, 1.0],
        #                 linestyles='dashed', colors=w_colors[i], alpha=0.3)
        cp1 = ax.contourf(X1,X2,G,1,linewidth=4, linestyles='dashed', alpha=0.8)
        ax.clabel(cp1, inline=True, fontsize=14)

        plt.colorbar(cp1)
        ax.set_title(titles[i])
        #ax.set_axis_off()  #ax.axis('off')
        #ax.axes.xaxis.set_ticks([])
        #ax.axes.yaxis.set_ticks([])
        ax.set_xlim(-p,p)
        ax.set_ylim(-p,p)
    fig.tight_layout()
```

```python
w_no_reg = train_with_regularization(0.0,X_train_all,y_train_all,use_sklearn=False)
w_reg    = train_with_regularization(0.1,X_train_all,y_train_all,use_sklearn=False)
E_in_no_reg,E_out_no_reg = get_E_in_E_out(X_test,y_test,X_train_all,y_train_all,w_no_reg)
E_in_reg,   E_out_reg    = get_E_in_E_out(X_test,y_test,X_train_all,y_train_all,w_reg)
```

```python
def string_Ein(E_in):
    return '$E_{in}=' + '{}'.format(np.round(E_in,5)) + '$'

def string_Eout(E_out):
    return '$E_{out}=' + '{}'.format(np.round(E_out,5)) + '$'
```

```python
w_arr = [ w_no_reg, w_reg ]
w_colors = ['red', 'blue' ]
title_train = ['Training Set: (a) $\mathbf{w}$, ' + string_Ein(E_in_no_reg),
               'Training Set: (b) $\mathbf{w_{reg}}$, ' + string_Ein(E_in_reg)]
title_test  = ['Testing Set: (a) $\mathbf{w}$, ' + string_Eout(E_out_no_reg),
               'Testing Set: (b) $\mathbf{w_{reg}}$, ' + string_Eout(E_out_reg)]
fig = plt.figure(figsize=(14,12))
plot_data_nonlinear(fig,[ 221, 222 ],X_train,y_train,w_arr,w_colors,title_train)
plot_data_nonlinear(fig,[ 223, 224 ],X_test, y_test, w_arr,w_colors,title_test)
```

## 3. Hypothesis sets

Consider the following hypothesis set

$$\mathcal{H}_Q = \left\{ h \; | \; h(x) = \mathbf{w^T z} = \sum\limits_{q=0}^Q w_q L_q(x) \right\}$$

We thus have $\mathcal{H_4, H_3, H_2, H_1}$ as:

$$\mathcal{H}_4 = \left\{ h \; | \; h(x) = \mathbf{w^T z} = \sum\limits_{q=0}^4 w_q L_q(x) \right\}$$

$$\mathcal{H}_3 = \left\{ h \; | \; h(x) = \mathbf{w^T z} = \sum\limits_{q=0}^3 w_q L_q(x) \right\}$$

$$\mathcal{H}_2 = \left\{ h \; | \; h(x) = \mathbf{w^T z} = \sum\limits_{q=0}^2 w_q L_q(x) \right\}$$

$$\mathcal{H}_1 = \left\{ h \; | \; h(x) = \mathbf{w^T z} = \sum\limits_{q=0}^1 w_q L_q(x) \right\}$$

We also have:

$$\mathcal{H}\left(Q, C, Q_0\right) = \left\{ h \; | \; h(x) = \mathbf{w^T z} \in \mathcal{H}_Q; w_Q = C \text{ for } q \ge Q_0 \right\}$$

For $Q = 10, C = 0, Q_0 = 3$:
$$\mathcal{H}\left(10, C, 3\right) = \left\{ h \; | \; h(x) = \mathbf{w^T z} \in \mathcal{H}_{10}; w_Q = 0 \text{ for } q \ge 3 \right\} = \mathcal{H}_2$$

For $Q = 10, C = 0, Q_0 = 4$:
$$\mathcal{H}\left(10, C, 4\right) = \left\{ h \; | \; h(x) = \mathbf{w^T z} \in \mathcal{H}_{10}; w_Q = 0 \text{ for } q \ge 4 \right\} = \mathcal{H}_3$$

Hence we have:

$$\mathcal{H}\left(10, C, 3\right) \cap \mathcal{H}\left(10, C, 4\right) = \mathcal{H}_2 \cap \mathcal{H}_3 = \mathcal{H}_2$$

## 4. Neural Networks

### Forward Propagation

$$x_j^{(l)} = \theta \left( s_j^{(l)} \right) = \theta \left( \sum\limits_{i=0}^{d^{(l-1)}} w_{ij}^{(l)} x_i^{(l-1)} \right)$$

where

$$\theta\left(s\right) = \tanh\left(s\right)$$

### Backpropagation

$$\delta w_{ij}^{(l)} = -\eta x_i^{(l-1)} \delta_j^{(l)}$$

where

$$\delta_j^{(l-1)} = \left[1 - \left(x_i^{(l-1)}\right)^2\right] \sum_{j=1}^{d^{(l)}} w_{ij}^{(l)} \delta_j^{(l)}$$

### Gradient Descent

$$\mathbf{w}\left(t+1\right) = \mathbf{w}\left(t\right) - \eta \nabla E_{in}\left(\mathbf{w}\left(t\right)\right) - 2\eta \frac{\lambda}{N} \mathbf{w}\left(t\right)$$

$$\mathbf{w}\left(t+1\right) = \mathbf{w}\left(t\right)\left(1 - 2\eta \frac{\lambda}{N}\right) - \eta \nabla E_{in}\left(\mathbf{w}\left(t\right)\right)$$

where

$$\mathbf{w^T w} = \sum\limits_{l=1}^L \sum\limits_{i=0}^{d^{(l-1)}} \sum\limits_{j=1}^{d^{(l)}} \left( w_{ij}^{(l)}\right)^2$$

Consider a neural network with 5 inputs, 3 nodes at the hidden layer, and a single node in the output layer, i.e. $d^{(0)} = 5, d^{(1)} = 3, d^{(2)} = 1$:

Further reading: http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html

```python
def calculate_weights(layer_nodes):
    """
    Calculate the number of weights required for a neural network,
    where layer_nodes contains the number of nodes for each layer,
    excluding bias units.
    """
    n = len(layer_nodes)
    total = 0
    for i in range(n-1):
        prev = layer_nodes[i]+1  # Add 1 to include bias unit
        curr = layer_nodes[i+1]
        total += prev*curr
        print("Layer {}: d+1={} -> {} = {}".format(i+1,prev,curr,prev*curr))
    return total
```

For a neural network with 5 inputs, 3 nodes in the hidden layer, and 1 node in the output layer, the total number of weights is calculated as follows:

```python
calculate_weights(layer_nodes = [5, 3, 1])
```

    Layer 1: d+1=6 -> 3 = 18
    Layer 2: d+1=4 -> 1 = 4

    22

Suppose we wish to calculate the minimum and maximum number of weights of a neural network with the restriction that it has 10 input units, 36 hidden units, and one output unit.  The minimum number of weights occurs when the 36 hidden units is split into 18 hidden layers (each with 1 bias node + 1 regular node).  The number of weights is calculated as follows:

```python
calculate_weights(layer_nodes = [9] + ([1] * 18) + [1])
```

    Layer 1: d+1=10 -> 1 = 10
    Layer 2: d+1=2 -> 1 = 2
    Layer 3: d+1=2 -> 1 = 2
    Layer 4: d+1=2 -> 1 = 2
    Layer 5: d+1=2 -> 1 = 2
    Layer 6: d+1=2 -> 1 = 2
    Layer 7: d+1=2 -> 1 = 2
    Layer 8: d+1=2 -> 1 = 2
    Layer 9: d+1=2 -> 1 = 2
    Layer 10: d+1=2 -> 1 = 2
    Layer 11: d+1=2 -> 1 = 2
    Layer 12: d+1=2 -> 1 = 2
    Layer 13: d+1=2 -> 1 = 2
    Layer 14: d+1=2 -> 1 = 2
    Layer 15: d+1=2 -> 1 = 2
    Layer 16: d+1=2 -> 1 = 2
    Layer 17: d+1=2 -> 1 = 2
    Layer 18: d+1=2 -> 1 = 2
    Layer 19: d+1=2 -> 1 = 2

    46

To calculate the maximum number of weights, a neural network with a single hidden layer would have the following number of weights:

```python
calculate_weights(layer_nodes = [9, 35, 1])
```

    Layer 1: d+1=10 -> 35 = 350
    Layer 2: d+1=36 -> 1 = 36

    386

If we had two hidden layers:

```python
calculate_weights([9, 17, 17, 1])
```

    Layer 1: d+1=10 -> 17 = 170
    Layer 2: d+1=18 -> 17 = 306
    Layer 3: d+1=18 -> 1 = 18

    494

```python
calculate_weights([9, 18, 16, 1])
```

    Layer 1: d+1=10 -> 18 = 180
    Layer 2: d+1=19 -> 16 = 304
    Layer 3: d+1=17 -> 1 = 17

    501

```python
calculate_weights([9, 19, 15, 1])
```

    Layer 1: d+1=10 -> 19 = 190
    Layer 2: d+1=20 -> 15 = 300
    Layer 3: d+1=16 -> 1 = 16

    506

```python
calculate_weights([9, 20, 14, 1])
```

    Layer 1: d+1=10 -> 20 = 200
    Layer 2: d+1=21 -> 14 = 294
    Layer 3: d+1=15 -> 1 = 15

    509

```python
calculate_weights([9, 21, 13, 1])
```

    Layer 1: d+1=10 -> 21 = 210
    Layer 2: d+1=22 -> 13 = 286
    Layer 3: d+1=14 -> 1 = 14

    510

If we had 3 hidden layers:

```python
calculate_weights([9, 11, 11, 11, 1])
```

    Layer 1: d+1=10 -> 11 = 110
    Layer 2: d+1=12 -> 11 = 132
    Layer 3: d+1=12 -> 11 = 132
    Layer 4: d+1=12 -> 1 = 12

    386

If we had 4 hidden layers:

```python
calculate_weights([9, 8, 8, 8, 8, 1])
```

    Layer 1: d+1=10 -> 8 = 80
    Layer 2: d+1=9 -> 8 = 72
    Layer 3: d+1=9 -> 8 = 72
    Layer 4: d+1=9 -> 8 = 72
    Layer 5: d+1=9 -> 1 = 9

    305

Hence, the maximum number of weights corresponds to the neural network [9, 21, 13, 1], with 10 units in the input layer, 22 units in the first hidden layer, and 14 units in the second hidden layer, and 1 unit in the final output layer.  The total number of weights for this network is 510.


# Скорости

Возьмем стандартные матрицы преобразования:

```python
from sympy import *
def rz(a):
    return Matrix([
        [cos(a), -sin(a), 0, 0],
        [sin(a), cos(a), 0, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1]
    ])

def ry(a):
    return Matrix([
        [cos(a), 0, sin(a), 0],
        [0, 1, 0, 0],
        [-sin(a), 0, cos(a), 0],
        [0, 0, 0, 1]
    ])

def rx(a):
    return Matrix([
        [1, 0, 0, 0],
        [0, cos(a), -sin(a), 0],
        [0, sin(a), cos(a), 0],
        [0, 0, 0, 1]
    ])

def trs(x, y, z):
    return Matrix([
        [1, 0, 0, x],
        [0, 1, 0, y],
        [0, 0, 1, z],
        [0, 0, 0, 1]
    ])

def vec(x, y, z):
    return Matrix([
        [x],
        [y],
        [z],
        [1]
    ])
```

Опишем двузвенный плоский механизм:

```python
q0, q1 = symbols("q_0, q_1", cls=Function)
t = symbols("t")
l0, l1 = symbols("l_0, l_1")
```

```python
pos = rz(q0(t)) * trs(l0, 0, 0) * rz(q1(t)) * trs(l1, 0, 0)
simplify(pos)
```

Продиффиринцируем положение такого механизма по оси $X$:

```python
vx = (pos * vec(0, 0, 0)).diff(t)[0]
simplify(vx)
```

И по оси $Y$:

```python
vy = (pos * vec(0, 0, 0)).diff(t)[1]
simplify(vy)
```

Тогда, из полученного результата можно вынести $\dot{q_0}$ и $\dot{q_1}$:

$$
V_x = -l_0 \sin(q_0) \dot{q_0} - l1 (\dot{q_0} + \dot{q_1}) \sin(q_0 + q_1)
$$
$$
V_x = -l_0 \sin(q_0) \cdot \dot{q_0} - l_1 \dot{q_0} \sin(q_0 + q_1) - l_1 \dot{q_1} \sin(q_0 + q_1)
$$

$$
V_x = \dot{q_0} (-l_0 \sin(q_0) - l_1 \sin(q_0 + q_1)) + \dot{q_1} (-l_1 \sin(q_0 + q_1))
$$
$$
V_y = \dot{q_0} (l_0 \cos(q_0) + l_1 \cos(q_0 + q_1)) + \dot{q_1} (l_1 \cos(q_0 + q_1))
$$

Заменим:

`Исходное` | замена
:---|---
`-l_0 \sin(q_0)` | $A$
`-l_1 \sin(q_0 + q_1)` | $B$
`l_0 \cos(q_0)` | $C$
`l_1 \cos(q_0 + q_1)` | $D$

$$
V_x = \dot{q_0} (A + B) + \dot{q_1} B
$$
$$
V_y = \dot{q_0} (C + D) + \dot{q_1} D
$$

Тогда такую запись можно предсавить в матричном виде:

$$
\begin{bmatrix}
    V_x \\
    V_y
\end{bmatrix} =
\begin{bmatrix}
    A + B & B \\
    C + D & D
\end{bmatrix}
\begin{bmatrix}
    \dot{q_0} \\
    \dot{q_1}
\end{bmatrix}
$$

Если учесть угловую скорость $\omega$ получим:

$$
\omega = \dot{q_0} + \dot{q_1}
$$

$$
\begin{bmatrix}
    V_x \\
    V_y \\
    \omega
\end{bmatrix} =
\begin{bmatrix}
    A + B & B \\
    C + D & D \\
    1 & 1
\end{bmatrix}
\begin{bmatrix}
    \dot{q_0} \\
    \dot{q_1}
\end{bmatrix}
=
J
\begin{bmatrix}
    \dot{q_0} \\
    \dot{q_1}
\end{bmatrix}
$$

Матрица $J$ называется _матрицей Якоби_ или _Якобианом_.
Она описывает взаимоотношение скорости обобщенных координат и скорости звена (например, фланца).


```python
%matplotlib notebook
from sympy import *
import numpy as np
import matplotlib.pyplot as plt
init_printing()
```

```python
var("x y")
```

```python
f=(x**2)+(y**2)
f
```

```python
dfx=f.diff(x)
dfy=f.diff(y)
pc=solve([dfx,dfy],[x,y])

dfx,dfy,pc
```

```python
dfxx=dfx.diff(x)
dfxy=dfx.diff(y)
dfyx=dfy.diff(x)
dfyy=dfy.diff(y)

D=dfxx*dfyy - dfxy
D.subs(pc),dfxx.subs(pc)
```

```python
#matriz con segunda derivada
H=hessian(f,[x,y])
H
```

$$\left[\begin{matrix}2 & 0\\0 & 2\end{matrix}\right]$$

```python
H.eigenvals()
```

```python
#ejemplo
```

```python
f=(x**3)+3*(x**2)+3*(y**3)+3*(y**2)+24
f
```

```python
#gradient
nablaf=[f.diff(var) for var in [x,y]]
nablaf
```

```python
#puntos criticos
pcs=solve(nablaf)
pcs
```

```python
H=hessian(f,[x,y])
H
```

$$\left[\begin{matrix}6 x + 6 & 0\\0 & 18 y + 6\end{matrix}\right]$$

```python
for pc in pcs:
    eig=H.subs(pc).eigenvals()
    print("Puntos criticos:",pc,"sus eigenvalores:",eig)
```

    Puntos criticos: {x: -2, y: -2/3} sus eigenvalores: {-6: 2}
    Puntos criticos: {x: -2, y: 0} sus eigenvalores: {-6: 1, 6: 1}
    Puntos criticos: {x: 0, y: -2/3} sus eigenvalores: {-6: 1, 6: 1}
    Puntos criticos: {x: 0, y: 0} sus eigenvalores: {6: 2}

```python
x_=np.arange(-3,3,.1)
y_=np.arange(-3,3,.1)

xx_, yy_=np.meshgrid(x_,y_)
F =xx_**3 + 3*yy_**3 + 3*xx_**2 + 3*yy_**2 + 24
```

```python
fig = plt.figure()
ax = fig.gca(projection="3d")
surf = ax.plot_surface(xx_,yy_,F)

ax.set_xlaber("X Label")
ax.set_ylaber("Y Label")
ax.set_zlaber("Z Label")
plt.show()
```

```python

```


# Non-uniform structured grid

In fluid mechanics and heat transfer, the solution often contains regions of large gradients and regions of small gradients. In cases where both regions are well localized, computational efficiency favors clustering computational nodes in regions of large gradients and mapping regions of small gradients with coarse meshes. This notebook described a common method to define a non-uniform grid for a symmetrical problem, with the finest resolution at the two ends of the domain, e.g. channel flow.

## Grid transformation

Consider a domain $[-H/2,+H/2]$ discretized with $N$ points. The location of computational nodes on a uniform grid is defined as:

$$
\tilde{y}_j = -h + j\Delta \text{ with }\Delta = \frac{2h}{N-1}\text{ and }j\in[0,N-1]
$$
where $h=H/2$.

The common approach to create a non-uniform grid is to operate a transform function over the uniform grid. For a channel flow, one such function is:

$$
y_j = h\frac{\tanh\gamma \tilde{y}_j}{\tanh\gamma h}
$$

The coefficient $\gamma$ controls the stretching of the grid, in this case the minimum mesh size, which is at the wall.

### Python set-up and useful functions

```python
%matplotlib inline
# plots graphs within the notebook

from IPython.display import display,Image, Latex
from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')
from IPython.display import clear_output

import time

import numericaltools as numtools

import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc
import h5py

import sympy as sym

class PDF(object):
  def __init__(self, pdf, size=(200,200)):
    self.pdf = pdf
    self.size = size

  def _repr_html_(self):
    return ''.format(self.pdf, self.size)

  def _repr_latex_(self):
    return r'\includegraphics[width=1.0\textwidth]{{{0}}}'.format(self.pdf)

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)

font = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 16,
        }
fontlabel = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 16,
        }

from matplotlib.ticker import FormatStrFormatter
plt.rc('font', **font)

```

## Example

Create a grid for $H=2$, $N=33$, and $\Delta_{min}=0.001$

```python
H = 2
N = 33
Deltamin = 0.0001
gamma_guess = 2
y,gamma = numtools.stretched_mesh(H,N,Deltamin,gamma_guess)
print("Stretching coefficient gamma: %1.4e" %gamma)
plt.plot(y,'o')
plt.show()
plt.plot(0.5*(y[1:]+y[:-1]),y[1:]-y[:-1],'o')
plt.show()

```

## Grid generation at fixed $\gamma$

$H=2$, $N=257$, $\gamma = 2.6$

```python
H = 2.
h = H/2
N =257
gamma = 2.6
y_uni = np.linspace(-h,h,N)
y = h*np.tanh(gamma*y_uni)/np.tanh(gamma*h)
plt.plot(y,'o')
plt.show()
plt.plot(0.5*(y[1:]+y[:-1]),y[1:]-y[:-1],'o')
plt.show()
print("Deltamin = %1.4e" %(y[1]-y[0]))
```

## Generate a bunch of grids with different $N$ but constant $\Delta_{min}$

```python
H = 2
deltamin = 5e-4
gamma_guess = 2.
N_array = np.array([33,65,129,257,513,1025],dtype=int)
gamma_array = np.zeros(len(N_array))
j = 0
for N in N_array:
    y,gamma_array[j] = numtools.stretched_mesh(H,N,Deltamin,gamma_guess)
    gamma_guess = gamma_array[j]
    print("for N=%4i, gamma=%1.4f" %(N,gamma_array[j]))

    j += 1

print(gamma_array)

```

    for N=  33, gamma=4.8624
    for N=  65, gamma=4.3730
    for N= 129, gamma=3.9348
    for N= 257, gamma=3.5145
    for N= 513, gamma=3.0970
    for N=1025, gamma=2.6734
    [4.86238469 4.37303735 3.93484222 3.51452064 3.09698058 2.67343703]

## Generate a bunch of grids at constant $N$ and varying $\Delta_{min}$

```python
# Deltamin_array = np.array([1e-2,5e-3,1e-3,5e-4,1e-4,5e-5,1e-5])
Deltamin_array = np.array([1e-3,5e-4,1e-4,5e-5,1e-5])
H = 2
N = 257
gamma_guess = 2
for Deltamin in Deltamin_array:
    y,gamma = numtools.stretched_mesh(H,N,Deltamin,gamma_guess)
    print("For Dmin=%1.1e, gamma=%1.4f" %(Deltamin,gamma))
```

    For Dmin=1.0e-03, gamma=2.1001
    For Dmin=5.0e-04, gamma=2.5444
    For Dmin=1.0e-04, gamma=3.5145
    For Dmin=5.0e-05, gamma=3.9169
    For Dmin=1.0e-05, gamma=4.8300

```python
2/129
```

$$0.015503875968992248$$

```python

```


## One Step Method
Applying the Euler formula to the first order equation
\begin{equation}
y^{'} = (1-x)y^2-y
\end{equation}
is approximated by

\begin{equation}
\frac{w_{i+1}-w_i}{h}=(1-x_i)w_i^2-w_i
\end{equation}

Rearranging
\begin{equation}
w_{i+1}=w_i+h((1-x_i)w_i^2-w_i)
\end{equation}

## DECLARING LIBRARIES

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
warnings.filterwarnings("ignore")
```

## Setting up X

```python
h=0.25
x_end=1
INITIALCONDITION=1
N=int(x_end/h)
Numerical_Solution=np.zeros(N+1)
x=np.zeros(N+1)

```

## NUMERICAL SOLUTION

```python
Numerical_Solution[0]=INITIALCONDITION
x[0]=0
Analytic_Solution[0]=INITIALCONDITION
for i in range (1,N+1):
    Numerical_Solution[i]=Numerical_Solution[i-1]+h*((1-x[i-1])*Numerical_Solution[i-1]*Numerical_Solution[i-1]-Numerical_Solution[i-1])
    x[i]=x[i-1]+h

```

## Plotting

```python
fig = plt.figure(figsize=(8,4))
plt.plot(x,Numerical_Solution,color='red')
#ax.legend(loc='best')
plt.title('Numerical Solution')
# --- right hand plot
print('x')
print(x)
print('Numerical Solutions')
print(Numerical_Solution)

```

```python

```

Chapter 01 - Euler Methods/Supplementary/06_Euler Method - Review Questions 1c.ipynb


# Tutorial rápido de Python para Matemáticos

© Ricardo Miranda Martins, 2022 - http://www.ime.unicamp.br/~rmiranda/

## Índice

1. [Introdução](1-intro.html)
2. **[Python é uma boa calculadora!](2-calculadora.html)** [(código fonte)](2-calculadora.ipynb)
3. [Resolvendo equações](3-resolvendo-eqs.html)  [(código fonte)](3-resolvendo-eqs.ipynb)
4. [Gráficos](4-graficos.html)  [(código fonte)](4-graficos.ipynb)
5. [Sistemas lineares e matrizes](5-lineares-e-matrizes.html)  [(código fonte)](5-lineares-e-matrizes.ipynb)
6. [Limites, derivadas e integrais](6-limites-derivadas-integrais.html)  [(código fonte)](6-limites-derivadas-integrais.ipynb)
7. [Equações diferenciais](7-equacoes-diferenciais.html)  [(código fonte)](7-equacoes-diferenciais.ipynb)

# Python é uma boa calculadora!

Vamos colocar o Python para fazer umas contas. Para isso, é só digitar o que você quer na linha aqui de baixo e depois apertar shift+enter. Tudo que você digitar na linha de código começando com # será interpretado como um comentário, e em geral é usado para explicar partes do código.

```python
# isto é um comentário. abaixo, fazemos a soma de 1 com 1
1+1
```

    2

```python
232323*8987
```

    2087886801

Para dividir dois números, é só usar o símbolo "/":

```python
11/4
```

    2.75

O resultado será um número decimal. Se você quiser fazer uma divisão inteira, aquela em que é produzido um quociente e um resto, terá que usar dois comandos. O ```//``` produz o quociente dessa divisão e o ```%``` o resto, como nos exemplos abaixo. Lembre-se que $11=4\cdot 2+3$.

```python
11//4
```

    2

```python
11 % 4
```

    3

O Python tem suas funções ampliadas com base em módulos/bibliotecas. Usaremos muitas delas em nossos exemplos.

Uma das bibliotecas mais poderosas para matemática é a SymPy, para cálculos simbólicos. A SymPy é uma biblioteca enorme, então se importamos todas as funções dela para o ambiente de trabalho, além de começar a dar conflito com algumas outras funções, tudo pode ficar mais lento. Então o comando abaixo vai importar SymPy usando um atalho. Esse é um procedimento totalmente usual. No [site do SymPy](https://docs.sympy.org/latest/tutorial/index.html) você tem um tutorial bem legal que eu recomendo.

O comando ```import sympy as sp``` diz que estamos importando a biblioteca SymPy e o prefixo ```sp``` será usado para chamar as funções dessa biblioteca. A seguir, usamos o comando ```isprime```, que determina se um número é ou não primo.

```python
import sympy as sp
sp.isprime(11)
```

    True

## Sem preguiça, vamos programar um pouco!

Usar a função ```isprime``` do SymPy é ótimo, mas poderíamos ter programado um "crivo" para verificar primalidade, né? Vamos fazer isso. Vai ser uma primeira oportunidade de conhecer a sintaxe de alguns laços, que podem ser úteis mais pra frente.

Uma forma simples de verificar se um número é primo é a seguinte: tente dividí-lo por todos os inteiros maiores que 1 e menores que ele. Se nenhuma divisão for exata, o número é primo. Se em alguma divisão der resto zero, então o número não será primo. O código abaixo foi adaptado [daqui](https://stackoverflow.com/questions/4114167/checking-if-a-number-is-prime-in-python).

```python
def checaprimo(n):
    if n == 2:
        return True
    if n % 2 == 0 or n <= 1:
        return False
    for divisor in range(3, n, 2):
        if n % divisor == 0:
            return False
    return True
checaprimo(101)
```

    True

Agora vamos pensar um pouco: estamos tentando dividir $n$ por todos os números entre $2$ e $n-1$. Será que precisamos de tudo isso? Ora, se $n$ for divisível por $k$, com $k<n$, então $n$ também será divisível por $n/k$. Das duas, uma: ou $k$ ou $n/k$ será menor que $\sqrt{n}$. A prova disso é fácil: se ambos forem maiores que $\sqrt{n}$, então o produto deles, que sabemos que resulta em $n$, seria $$n=k\cdot (n/k)>\sqrt{n}\cdot \sqrt{n}=n,$$ um absurdo portanto.

Logo, podemos alterar o ```range``` dentro do ```for``` para ir de $3$ até o inteiro maior ou igual a$\sqrt{n}$ (repare no uso da função ```int``` no código).

Como a função $\sqrt{x}$ não está implementada por padrão no Python, ao invés de implementá-la, vamos carregar a biblioteca ```math``` que vem com ela e outras funções simples no pacote. Veja detalhes sobre a ```math``` [aqui](https://docs.python.org/3/library/math.html).

Note que isso aumenta bastante a velocidade de execução do nosso algoritmo.

```python
import math
def checaprimo2(n):
    if n == 2:
        return True
    if n % 2 == 0 or n <= 1:
        return False
    for divisor in range(3, int(math.sqrt(n)), 2):
        if n % divisor == 0:
            return False
    return True
checaprimo2(101)
```

    True

## Bônus: programando nossa própria raiz quadrada

No código acima, para verificar a primalidade, usamos a função ```sqrt``` do pacote ```math``` do Python, mas isso é um exagero. Podemos facilmente programar uma função que calcula a raiz quadrada.

A função que vamos programar usa um argumento simples, que era usado pelos babilônios por volta do ano 60 d.C. para calcular a raiz quadrada: suponha que $x>0$ e queremos calcular $\sqrt{x}$. Seja $y$ com $y^2<x$. Então $\sqrt{x}$ está entre $y$ e $x/y$ ("perto do ponto médio desse intervalo"). Atualmente essa estratégia é conhecida como método de Heron, e sua demonstração é simples (tem até [aqui](https://en.wikipedia.org/wiki/Methods_of_computing_square_roots#Babylonian_method) na Wikipedia).

Em termos mais algoritmicos, a ideia é a seguinte. Queremos calcular $\sqrt{x}$. Escolha $y=1$ (ou qualquer outro número positivo). Então, a menos que $x=1$, a raiz quadrada de $x$ estará entre $y$ e $x/y$. Pegue então o ponto médio desse intervalo e chame de $y_1$. Novamente, a raiz quadrada de $x$ estará entre $y_1$ e $x/y_1$. Pegue novamente o ponto médio desse intervalo e continue sucessivamente. A sequência desses "pontos médios" vai convergir para a raiz quadrada.

```python
def minharaizquadrada(x):
    y=1
    # aqui é que definimos o numero de iterações, troque 10 pelo que quiser;
    # quando maior, melhor aproximação
    for k in range(1,10):
        y = (y + x/y)/2
    return y

# calculando sqrt(2)
minharaizquadrada(2)
```

    1.414213562373095

Pronto, agora temos nossa própria função para calcular raiz quadrada! Vamos rodar novamente aquele teste de primalidade, só que agora com nossa função raiz quadrada.

```python
def checaprimo3(n):
    if n == 2:
        return True
    if n % 2 == 0 or n <= 1:
        return False
    for divisor in range(3, int(minharaizquadrada(n)), 2):
        if n % divisor == 0:
            return False
    return True
checaprimo3(101)
```

    True

Bom, 101 continua sendo primo, então a função deve funcionar :)


# Graph Attention Networks (GATs)
Original paper: Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y. (2017). [Graph attention networks](https://arxiv.org/abs/1710.10903). *arXiv preprint arXiv:1710.10903*.

## Math Warm-up
This section comes from https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html, with minor extensions.

### GCN Layer
$$h_i^{(l+1)}=\sigma\left(\sum_{j\in \mathcal{N}(i)} {\frac{1}{c_{ij}} W^{(l)}h^{(l)}_j}\right)$$

* $\mathcal{N}(i)$: set of the one-hop neighbors (no self-loop) of $n_i$.
* $c_{ij}=\sqrt{|\mathcal{N}(i)|}\sqrt{|\mathcal{N}(j)|}$: normalization costant based on the graph structure.
* $\sigma$: activation function.
* $W^{(l)}$: weight matrix for feature transformation.

A broad explanation on Graph Neural Networks (GNNs) is available in the Medium article entitled "[Understanding the Building Blocks of Graph Neural Networks](https://towardsdatascience.com/understanding-the-building-blocks-of-graph-neural-networks-intro-56627f0719d5)".

### GAT Layer

\begin{split}\begin{align}
z_i^{(l)}&=W^{(l)}h_i^{(l)}, \\
e_{ij}^{(l)}&=\text{LeakyReLU}(\vec a^{(l)^T}(z_i^{(l)}||z_j^{(l)})),\\
\alpha_{ij}^{(l)}&=\frac{\exp(e_{ij}^{(l)})}{\sum_{k\in \mathcal{N}(i)}^{}\exp(e_{ik}^{(l)})},\\
h_i^{(l+1)}&=\sigma\left(\sum_{j\in \mathcal{N}(i)} {\alpha^{(l)}_{ij} z^{(l)}_j }\right),
\end{align}\end{split}

* Equation (1) is a linear transformation of the lower layer embedding $h_i^{(l)}$ and $W^{(l)}$ is its learnable weight matrix. This transformation is useful to achieve a sufficient expressive power to transform input features (in our example one-hot vectors) into high-level and dense features.
* Equation (2) computes a pair-wise *un-normalized* attention score between two neighbors. Here, it first concatenates the $z$ embeddings of the two nodes, where $||$ denotes concatenation, then takes a dot product of it and a learnable weight vector $\vec a^{(l)}$, and applies a LeakyReLU in the end. This form of attention is usually called additive attention, contrast with the dot-product attention in the Transformer model. The attention score indicates the importance of a neighbor node in the message passing framework.
* Equation (3) applies a softmax to normalize the attention scores on each node’s incoming edges.
* Equation (4) is similar to GCN. The embeddings from neighbors are aggregated together, scaled by the attention scores.

## Imports

```python
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import dgl

np.random.seed(1)

```

    Using backend: pytorch

## GAT Layer Implementation with NumPy
### Basic Functions

```python
def leaky_relu(z):
    return np.where(z > 0, z, z * 0.01)

def softmax(z):
    if len(z.shape) > 1:
        # Softmax for matrix
        max_matrix = np.max(z, axis=0)
        stable_z = z - max_matrix
        e = np.exp(stable_z)
        a = e / np.sum(e, axis=0, keepdims=True)
    else:
        # Softmax for vector
        vector_max_value = np.max(z)
        a = (np.exp(z - vector_max_value)) / sum(np.exp(z - vector_max_value))

    assert a.shape == z.shape

    return a

```

### Graph and Weight Matrix Generation

```python
print('\n\n----- One-hot vector representation of nodes. Shape(n,n)\n')
X = np.eye(5, 5)
n = X.shape[0]
np.random.shuffle(X)
print(X)

print('\n\n----- Embedding dimension\n')
emb = 3
print(emb)

print('\n\n----- Weight Matrix. Shape(emb, n)\n')
W = np.random.uniform(-np.sqrt(1. / emb), np.sqrt(1. / emb), (emb, n))
print(W)

print('\n\n----- Adjacency Matrix (undirected graph). Shape(n,n)\n')
A = np.random.randint(2, size=(n, n))
np.fill_diagonal(A, 1)
A = (A + A.T)
A[A > 1] = 1
print(A)
```

    ----- One-hot vector representation of nodes. Shape(n,n)

    [[0. 0. 1. 0. 0.]
     [0. 1. 0. 0. 0.]
     [0. 0. 0. 0. 1.]
     [1. 0. 0. 0. 0.]
     [0. 0. 0. 1. 0.]]

    ----- Embedding dimension

    3

    ----- Weight Matrix. Shape(emb, n)

    [[-0.4294049   0.57624235 -0.3047382  -0.11941829 -0.12942953]
     [ 0.19600584  0.5029172   0.3998854  -0.21561317  0.02834577]
     [-0.06529497 -0.31225734  0.03973776  0.47800217 -0.04941563]]

    ----- Adjacency Matrix (undirected graph). Shape(n,n)

    [[1 1 1 0 1]
     [1 1 1 1 1]
     [1 1 1 1 0]
     [0 1 1 1 1]
     [1 1 0 1 1]]

### Linear Transformation

```python
# equation (1)
print('\n\n----- Linear Transformation. Shape(n, emb)\n')
z1 = X.dot(W.T)
print(z1)
```

    ----- Linear Transformation. Shape(n, emb)

    [[-0.3047382   0.3998854   0.03973776]
     [ 0.57624235  0.5029172  -0.31225734]
     [-0.12942953  0.02834577 -0.04941563]
     [-0.4294049   0.19600584 -0.06529497]
     [-0.11941829 -0.21561317  0.47800217]]

### Transformer: Additive Attention Mechanism

```python
# equation (2)
print('\n\n----- Concat hidden features to represent edges. Shape(len(emb.concat(emb)), number of edges)\n')
edge_coords = np.where(A==1)
h_src_nodes = z1[edge_coords[0]]
h_dst_nodes = z1[edge_coords[1]]
z2 = np.concatenate((h_src_nodes, h_dst_nodes), axis=1)

# Concatenation tests
assert len(edge_coords[1]) == z2.shape[0], "The number of edges in A is not equal to the number of concat edges"
test_value = np.array([-0.11941829, -0.12942953, 0.19600584, 0.5029172, 0.3998854, -0.21561317])
assert z2[4 ,:].tolist().sort()  == test_value.tolist().sort(), "Something went wrong in the concat process"
print(z2)

print('\n\n----- Attention coefficients. Shape(1, len(emb.concat(emb)))\n')
att = np.random.rand(1, z2.shape[1])
print(att)

print('\n\n----- Edge representations combined with the attention coefficients. Shape(1, number of edges)\n')
z2_att = z2.dot(att.T)
print(z2_att)

print('\n\n----- Leaky Relu. Shape(1, number of edges)')
e = leaky_relu(z2_att)
print(e)
```

    ----- Concat hidden features to represent edges. Shape(len(emb.concat(emb)), number of edges)

    [[-0.3047382   0.3998854   0.03973776 -0.3047382   0.3998854   0.03973776]
     [-0.3047382   0.3998854   0.03973776  0.57624235  0.5029172  -0.31225734]
     [-0.3047382   0.3998854   0.03973776 -0.12942953  0.02834577 -0.04941563]
     [-0.3047382   0.3998854   0.03973776 -0.11941829 -0.21561317  0.47800217]
     [ 0.57624235  0.5029172  -0.31225734 -0.3047382   0.3998854   0.03973776]
     [ 0.57624235  0.5029172  -0.31225734  0.57624235  0.5029172  -0.31225734]
     [ 0.57624235  0.5029172  -0.31225734 -0.12942953  0.02834577 -0.04941563]
     [ 0.57624235  0.5029172  -0.31225734 -0.4294049   0.19600584 -0.06529497]
     [ 0.57624235  0.5029172  -0.31225734 -0.11941829 -0.21561317  0.47800217]
     [-0.12942953  0.02834577 -0.04941563 -0.3047382   0.3998854   0.03973776]
     [-0.12942953  0.02834577 -0.04941563  0.57624235  0.5029172  -0.31225734]
     [-0.12942953  0.02834577 -0.04941563 -0.12942953  0.02834577 -0.04941563]
     [-0.12942953  0.02834577 -0.04941563 -0.4294049   0.19600584 -0.06529497]
     [-0.4294049   0.19600584 -0.06529497  0.57624235  0.5029172  -0.31225734]
     [-0.4294049   0.19600584 -0.06529497 -0.12942953  0.02834577 -0.04941563]
     [-0.4294049   0.19600584 -0.06529497 -0.4294049   0.19600584 -0.06529497]
     [-0.4294049   0.19600584 -0.06529497 -0.11941829 -0.21561317  0.47800217]
     [-0.11941829 -0.21561317  0.47800217 -0.3047382   0.3998854   0.03973776]
     [-0.11941829 -0.21561317  0.47800217  0.57624235  0.5029172  -0.31225734]
     [-0.11941829 -0.21561317  0.47800217 -0.4294049   0.19600584 -0.06529497]
     [-0.11941829 -0.21561317  0.47800217 -0.11941829 -0.21561317  0.47800217]]

    ----- Attention coefficients. Shape(1, len(emb.concat(emb)))

    [[0.09834683 0.42110763 0.95788953 0.53316528 0.69187711 0.31551563]]

    ----- Edge representations combined with the attention coefficients. Shape(1, number of edges)

    [[ 0.30322275]
     [ 0.73315639]
     [ 0.11150219]
     [ 0.11445879]
     [ 0.09607946]
     [ 0.52601309]
     [-0.0956411 ]
     [-0.14458757]
     [-0.0926845 ]
     [ 0.07860653]
     [ 0.50854017]
     [-0.11311402]
     [-0.16206049]
     [ 0.53443082]
     [-0.08722337]
     [-0.13616985]
     [-0.08426678]
     [ 0.48206613]
     [ 0.91199976]
     [ 0.2413991 ]
     [ 0.29330217]]

    ----- Leaky Relu. Shape(1, number of edges)
    [[ 3.03222751e-01]
     [ 7.33156386e-01]
     [ 1.11502195e-01]
     [ 1.14458791e-01]
     [ 9.60794571e-02]
     [ 5.26013092e-01]
     [-9.56410988e-04]
     [-1.44587571e-03]
     [-9.26845030e-04]
     [ 7.86065337e-02]
     [ 5.08540169e-01]
     [-1.13114022e-03]
     [-1.62060495e-03]
     [ 5.34430817e-01]
     [-8.72233739e-04]
     [-1.36169846e-03]
     [-8.42667781e-04]
     [ 4.82066128e-01]
     [ 9.11999763e-01]
     [ 2.41399100e-01]
     [ 2.93302168e-01]]

### Normalize the Attention Scores

```python
# equation (3)
print('\n\n----- Edge scores as matrix. Shape(n,n)\n')
e_matr = np.zeros(A.shape)
e_matr[edge_coords[0], edge_coords[1]] = e.reshape(-1,)
print(e_matr)

print('\n\n----- For each node, normalize the edge (or neighbor) contributions using softmax\n')
alpha0 = softmax(e_matr[:,0][e_matr[:,0] != 0])
alpha1 = softmax(e_matr[:,1][e_matr[:,1] != 0])
alpha2 = softmax(e_matr[:,2][e_matr[:,2] != 0])
alpha3 = softmax(e_matr[:,3][e_matr[:,3] != 0])
alpha4 = softmax(e_matr[:,4][e_matr[:,4] != 0])
alpha = np.concatenate((alpha0, alpha1, alpha2, alpha3, alpha4))
print(alpha)

print('\n\n----- Normalized edge score matrix. Shape(n,n)\n')
A_scaled = np.zeros(A.shape)
A_scaled[edge_coords[0], edge_coords[1]] = alpha.reshape(-1,)
print(A_scaled)
```

    ----- Edge scores as matrix. Shape(n,n)

    [[ 3.03222751e-01  7.33156386e-01  1.11502195e-01  0.00000000e+00
       1.14458791e-01]
     [ 9.60794571e-02  5.26013092e-01 -9.56410988e-04 -1.44587571e-03
      -9.26845030e-04]
     [ 7.86065337e-02  5.08540169e-01 -1.13114022e-03 -1.62060495e-03
       0.00000000e+00]
     [ 0.00000000e+00  5.34430817e-01 -8.72233739e-04 -1.36169846e-03
      -8.42667781e-04]
     [ 4.82066128e-01  9.11999763e-01  0.00000000e+00  2.41399100e-01
       2.93302168e-01]]

    ----- For each node, normalize the edge (or neighbor) contributions using softmax

    [0.26263543 0.21349717 0.20979916 0.31406823 0.21610715 0.17567419
     0.1726313  0.1771592  0.25842816 0.27167844 0.24278118 0.24273876
     0.24280162 0.23393014 0.23388927 0.23394984 0.29823075 0.25138555
     0.22399017 0.22400903 0.30061525]

    ----- Normalized edge score matrix. Shape(n,n)

    [[0.26263543 0.21349717 0.20979916 0.         0.31406823]
     [0.21610715 0.17567419 0.1726313  0.1771592  0.25842816]
     [0.27167844 0.24278118 0.24273876 0.24280162 0.        ]
     [0.         0.23393014 0.23388927 0.23394984 0.29823075]
     [0.25138555 0.22399017 0.         0.22400903 0.30061525]]

### Neighborhood Diffusion (GCN) Scaled by the Attention Scores (GAT)

```python
# equation (4)
print('\n\nNeighborhood aggregation (GCN) scaled with attention scores (GAT). Shape(n, emb)\n')
ND_GAT = A_scaled.dot(z1)
print(ND_GAT)
```

    Neighborhood aggregation (GCN) scaled with attention scores (GAT). Shape(n, emb)

    [[-0.02166863  0.15062515  0.08352843]
     [-0.09390287  0.15866476  0.05716299]
     [-0.07856777  0.28521023 -0.09286313]
     [-0.03154513  0.10583032  0.04267501]
     [-0.07962369  0.19226439  0.069115  ]]

## GAT Layer - DGL Test
Original layer implementation: https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html

```python
class GATTestLayer(nn.Module):
    def __init__(self, g, in_dim, out_dim):
        super(GATTestLayer, self).__init__()
        self.g = g
        # equation (1)
        self.fc = nn.Linear(in_dim, out_dim, bias=False)
        # equation (2)
        self.attn_fc = nn.Linear(2 * out_dim, 1, bias=False)
        self.reset_parameters()

    def reset_parameters(self):
        """Reinizialitation modified for testing"""
        gain = nn.init.calculate_gain('relu')
        self.fc.state_dict()['weight'][:] = torch.from_numpy(W)
        self.attn_fc.state_dict()['weight'][:] = torch.from_numpy(att)

    def edge_attention(self, edges):
        # edge UDF for equation (2)
        z2 = torch.cat([edges.src['z'], edges.dst['z']], dim=1)
        a = self.attn_fc(z2)
        return {'e': F.leaky_relu(a)}

    def message_func(self, edges):
        # message UDF for equation (3) & (4)
        return {'z': edges.src['z'], 'e': edges.data['e']}

    def reduce_func(self, nodes):
        # reduce UDF for equation (3) & (4)
        # equation (3)
        alpha = F.softmax(nodes.mailbox['e'], dim=1)
        # equation (4)
        h = torch.sum(alpha * nodes.mailbox['z'], dim=1)
        return {'h': h}

    def forward(self, h):
        # equation (1)
        z = self.fc(h)
        self.g.ndata['z'] = z
        # equation (2)
        self.g.apply_edges(self.edge_attention)
        # equation (3) & (4)
        self.g.update_all(self.message_func, self.reduce_func)
        return self.g.ndata.pop('h')
```

```python
print('\n\n----- Create a new DGL graph using the NumPy graph\n')
src_ids = torch.tensor(edge_coords[0])
dst_ids = torch.tensor(edge_coords[1])
g = dgl.graph((src_ids, dst_ids))
print(g)

print('\n\n----- Create a DGL instance of the GAT test layer\n')
net = GATTestLayer(g,
          in_dim=n,
          out_dim=3)
print(net.forward(torch.Tensor(X)))

print('\n\n----- Recap of the NumPy GAT layer')
print(np.round(ND_GAT, decimals=4))

```

    ----- Create a new DGL graph using the NumPy graph

    Graph(num_nodes=5, num_edges=21,
          ndata_schemes={}
          edata_schemes={})

    ----- Create a DGL instance of the GAT test layer

    tensor([[-0.0217,  0.1506,  0.0835],
            [-0.0939,  0.1587,  0.0572],
            [-0.0786,  0.2852, -0.0929],
            [-0.0315,  0.1058,  0.0427],
            [-0.0796,  0.1923,  0.0691]], grad_fn=<IndexCopyBackward>)

    ----- Recap of the NumPy GAT layer
    [[-0.0217  0.1506  0.0835]
     [-0.0939  0.1587  0.0572]
     [-0.0786  0.2852 -0.0929]
     [-0.0315  0.1058  0.0427]
     [-0.0796  0.1923  0.0691]]

The resulting matrices from the NumPy implementation and the DGL implementation are equal \o/.

## Math Warm-up on Multi-head Attention
The multi-head attention is useful to enrich the model capability and to stabilize the learning process. The outputs of each attention head can be combined in two different ways:

$\text{concatenation}: h^{(l+1)}_{i} =||_{k=1}^{K}\sigma\left(\sum_{j\in \mathcal{N}(i)}\alpha_{ij}^{k}W^{k}h^{(l)}_{j}\right)$

or

$\text{average}: h_{i}^{(l+1)}=\sigma\left(\frac{1}{K}\sum_{k=1}^{K}\sum_{j\in\mathcal{N}(i)}\alpha_{ij}^{k}W^{k}h^{(l)}_{j}\right)$

* K is the number of heads. Concatenation is adopted for intermediary layers. The average is employed for the final (prediction) layer, because the concatenation is no longer sensible.

## Multi Head GAT Layer Implementation with NumPy
Multiple head attentions are created generating multiple GAT layers.

```python
print('\n\n----- Recap on the output of the GAT layer')
print('\nLayer 1. Shape(emb,n)')
layer1 = ND_GAT
print(layer1)

print('\nLayer 2. Shape(emb,n)')
layer2 = ND_GAT
print(layer2)

print('\n\n----- Concatenate multiple attentions. Shape(num_layers*emb, n)\n')
concat = np.concatenate((layer1, layer2), axis=1)
print(concat)

print('\n\n----- Average multiple attentions.\n')
# 30 is the number of parameters: num_layers*emb*n
average = np.sum((layer1, layer2)) / 30
print(average)
```

    ----- Recap on the output of the GAT layer

    Layer 1. Shape(emb,n)
    [[-0.02166863  0.15062515  0.08352843]
     [-0.09390287  0.15866476  0.05716299]
     [-0.07856777  0.28521023 -0.09286313]
     [-0.03154513  0.10583032  0.04267501]
     [-0.07962369  0.19226439  0.069115  ]]

    Layer 2. Shape(emb,n)
    [[-0.02166863  0.15062515  0.08352843]
     [-0.09390287  0.15866476  0.05716299]
     [-0.07856777  0.28521023 -0.09286313]
     [-0.03154513  0.10583032  0.04267501]
     [-0.07962369  0.19226439  0.069115  ]]

    ----- Concatenate multiple attentions. Shape(num_layers*emb, n)

    [[-0.02166863  0.15062515  0.08352843 -0.02166863  0.15062515  0.08352843]
     [-0.09390287  0.15866476  0.05716299 -0.09390287  0.15866476  0.05716299]
     [-0.07856777  0.28521023 -0.09286313 -0.07856777  0.28521023 -0.09286313]
     [-0.03154513  0.10583032  0.04267501 -0.03154513  0.10583032  0.04267501]
     [-0.07962369  0.19226439  0.069115   -0.07962369  0.19226439  0.069115  ]]

    ----- Average multiple attentions.

    0.04979367027023359

## Multi Head GAT Layer - DGL Test
Original layer implementation: https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html

```python
class MultiHeadGATTestLayer(nn.Module):
    def __init__(self, g, in_dim, out_dim, num_heads, merge='cat'):
        super(MultiHeadGATTestLayer, self).__init__()
        self.heads = nn.ModuleList()
        for i in range(num_heads):
            # Use the test layer for consistency with the NumPy implementation
            self.heads.append(GATTestLayer(g, in_dim, out_dim))
        self.merge = merge

    def forward(self, h):
        head_outs = [attn_head(h) for attn_head in self.heads]
        if self.merge == 'cat':
            # concat on the output feature dimension (dim=1)
            return torch.cat(head_outs, dim=1)
        else:
            # merge using average
            return torch.mean(torch.stack(head_outs))

print('\n\n----- Multi head GAT layer (concat operation). Shape(num_layers*emb, n)\n')
concat_net = MultiHeadGATTestLayer(g, in_dim=n, out_dim=3, num_heads=2)
print(concat_net)
print('\n----- DGL concat output\n')
print(concat_net.forward(torch.Tensor(X)))

print('\n----- Recap of the NumPy concatenation\n')
print(np.round(concat, decimals=4))

print('\n\n----- Multi head GAT Layer (average operation). Shape(emb, n)\n')
mean_net = MultiHeadGATTestLayer(g, in_dim=n, out_dim=3, num_heads=2, merge='mean')
print(mean_net)
print('\n----- DGL average output\n')
print(mean_net.forward(torch.Tensor(X)))

print('\n----- Recap of the NumPy average\n')
print(np.round(average, decimals=4))
```

    ----- Multi head GAT layer (concat operation). Shape(num_layers*emb, n)

    MultiHeadGATTestLayer(
      (heads): ModuleList(
        (0): GATTestLayer(
          (fc): Linear(in_features=5, out_features=3, bias=False)
          (attn_fc): Linear(in_features=6, out_features=1, bias=False)
        )
        (1): GATTestLayer(
          (fc): Linear(in_features=5, out_features=3, bias=False)
          (attn_fc): Linear(in_features=6, out_features=1, bias=False)
        )
      )
    )

    ----- DGL concat output

    tensor([[-0.0217,  0.1506,  0.0835, -0.0217,  0.1506,  0.0835],
            [-0.0939,  0.1587,  0.0572, -0.0939,  0.1587,  0.0572],
            [-0.0786,  0.2852, -0.0929, -0.0786,  0.2852, -0.0929],
            [-0.0315,  0.1058,  0.0427, -0.0315,  0.1058,  0.0427],
            [-0.0796,  0.1923,  0.0691, -0.0796,  0.1923,  0.0691]],
           grad_fn=<CatBackward>)

    ----- Recap of the NumPy concatenation

    [[-0.0217  0.1506  0.0835 -0.0217  0.1506  0.0835]
     [-0.0939  0.1587  0.0572 -0.0939  0.1587  0.0572]
     [-0.0786  0.2852 -0.0929 -0.0786  0.2852 -0.0929]
     [-0.0315  0.1058  0.0427 -0.0315  0.1058  0.0427]
     [-0.0796  0.1923  0.0691 -0.0796  0.1923  0.0691]]

    ----- Multi head GAT Layer (average operation). Shape(emb, n)

    MultiHeadGATTestLayer(
      (heads): ModuleList(
        (0): GATTestLayer(
          (fc): Linear(in_features=5, out_features=3, bias=False)
          (attn_fc): Linear(in_features=6, out_features=1, bias=False)
        )
        (1): GATTestLayer(
          (fc): Linear(in_features=5, out_features=3, bias=False)
          (attn_fc): Linear(in_features=6, out_features=1, bias=False)
        )
      )
    )

    ----- DGL average output

    tensor(0.0498, grad_fn=<MeanBackward0>)

    ----- Recap of the NumPy average

    0.0498

The resulting matrices from the NumPy implementation and the DGL implementation are equal \o/.

# From Theory to Practice
After the understanding of math and the implementation of GAT building blocks, we can run some experiments as reported in the original paper. Let's recap the DGL modules using a fair parameter initialization. The following implementation is based on the example available here: https://docs.dgl.ai/en/0.4.x/tutorials/models/1_gnn/9_gat.html.

## New Imports

```python
import time
from dgl import DGLGraph
from dgl.data import citation_graph as citegrh
import networkx as nx
```

## GAT Implementation with DGL

```python
class GATLayer(nn.Module):
    def __init__(self, g, in_dim, out_dim):
        super(GATLayer, self).__init__()
        self.g = g
        # equation (1)
        self.fc = nn.Linear(in_dim, out_dim, bias=False)
        # equation (2)
        self.attn_fc = nn.Linear(2 * out_dim, 1, bias=False)
        self.reset_parameters()

    def reset_parameters(self):
        """Reinitialize learnable parameters."""
        gain = nn.init.calculate_gain('relu')
        nn.init.xavier_normal_(self.fc.weight, gain=gain)
        nn.init.xavier_normal_(self.attn_fc.weight, gain=gain)

    def edge_attention(self, edges):
        # edge UDF for equation (2)
        z2 = torch.cat([edges.src['z'], edges.dst['z']], dim=1)
        a = self.attn_fc(z2)
        return {'e': F.leaky_relu(a)}

    def message_func(self, edges):
        # message UDF for equation (3) & (4)
        return {'z': edges.src['z'], 'e': edges.data['e']}

    def reduce_func(self, nodes):
        # reduce UDF for equation (3) & (4)
        # equation (3)
        alpha = F.softmax(nodes.mailbox['e'], dim=1)
        # equation (4)
        h = torch.sum(alpha * nodes.mailbox['z'], dim=1)
        return {'h': h}

    def forward(self, h):
        # equation (1)
        z = self.fc(h)
        self.g.ndata['z'] = z
        # equation (2)
        self.g.apply_edges(self.edge_attention)
        # equation (3) & (4)
        self.g.update_all(self.message_func, self.reduce_func)
        return self.g.ndata.pop('h')
```

```python
class MultiHeadGATLayer(nn.Module):
    def __init__(self, g, in_dim, out_dim, num_heads, merge='cat'):
        super(MultiHeadGATLayer, self).__init__()
        self.heads = nn.ModuleList()
        for i in range(num_heads):
            self.heads.append(GATLayer(g, in_dim, out_dim))
        self.merge = merge

    def forward(self, h):
        head_outs = [attn_head(h) for attn_head in self.heads]
        if self.merge == 'cat':
            # concat on the output feature dimension (dim=1)
            return torch.cat(head_outs, dim=1)
        else:
            # merge using average
            return torch.mean(torch.stack(head_outs))
```

```python
class GAT(nn.Module):
    def __init__(self, g, in_dim, hidden_dim, out_dim, num_heads):
        super(GAT, self).__init__()
        self.layer1 = MultiHeadGATLayer(g, in_dim, hidden_dim, num_heads)
        # Be aware that the input dimension is hidden_dim*num_heads since
        # multiple head outputs are concatenated together. Also, only
        # one attention head in the output layer.
        self.layer2 = MultiHeadGATLayer(g, hidden_dim * num_heads, out_dim, 1)

    def forward(self, h):
        h = self.layer1(h)
        h = F.elu(h)
        h = self.layer2(h)
        return h
```

## Evaluation Functions

```python
def accuracy(logits, labels):
    _, indices = torch.max(logits, dim=1)
    correct = torch.sum(indices == labels)
    return correct.item() * 1.0 / len(labels)

def evaluate(model, features, labels, mask):
    model.eval()
    with torch.no_grad():
        logits = model(features)
        logits = logits[mask]
        labels = labels[mask]
        return accuracy(logits, labels)
```

## Load Cora Dataset

```python
def load_cora_data():
    data = citegrh.load_cora()
    features = torch.FloatTensor(data.features)
    labels = torch.LongTensor(data.labels)
    train_mask = torch.BoolTensor(data.train_mask)
    val_mask = torch.BoolTensor(data.val_mask)
    test_mask = torch.BoolTensor(data.test_mask)
    g = DGLGraph(data.graph)
    return g, features, labels, train_mask, val_mask, test_mask

g, features, labels, train_mask, val_mask, test_mask = load_cora_data()
print('\n\n----- Features of CORA dataset')

print('\n----- Graph:')
print(g)

print('\n----- Features:')
print(features)
print(features.nonzero(as_tuple=True)[1])

print('\n----- Labels:')
print(labels)
print(labels.size())
output = torch.unique(labels)
occs = torch.bincount(labels)
print('----- Number of unique labels:')
print(output)
print('----- Number of label occurrences:')
print(occs)

print('\n----- Training mask:')
train_long = train_mask.long()
occs = torch.bincount(train_long)
print(output)
print(occs)

print('\n----- Validation mask:')
val_long = val_mask.long()
occs = torch.bincount(val_long)
print(output)
print(occs)

print('\n----- Testing mask:')
test_long = test_mask.long()
occs = torch.bincount(test_long)
print(output)
print(occs)

```

Analyzing the cora dataset, you can get the following information:

1. Nodes have no features (one-hot encoding vectors)
2. Node labels are uniformly distributed

## Training Loop

```python
# create the model, 2 heads, each head has hidden size 8
model = GAT(g,
          in_dim=features.size()[1],
          hidden_dim=8,
          out_dim=7,
          num_heads=2)

# create optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

# main loop
dur = []
for epoch in range(300):
    t0 = time.time()

    logits = model(features)
    logp = F.log_softmax(logits, 1)
    loss = F.nll_loss(logp[train_mask], labels[train_mask])

    train_acc = accuracy(logp[train_mask], labels[train_mask])

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    dur.append(time.time() - t0)

    print("Epoch {:05d} | Time(s) {:.4f} | Loss {:.4f} | Training Accuracy {:.4f}".format(
        epoch, np.mean(dur), loss.item(), train_acc))

    if epoch % 30==0:
        print("\nEval on validation dataset...")
        val_acc = evaluate(model, features, labels, val_mask)
        print("Validation Accuracy: {:.4f}\n".format(val_acc))

print()
acc = evaluate(model, features, labels, test_mask)
print("Test Accuracy {:.4f}".format(acc))

```

```python

```